Computer Use: When Claude Operates the Desktop
On October 22, 2024, Anthropic released computer use in public beta: Claude 3.5 Sonnet operates desktops via screenshots, mouse, and keyboard. We explain the agent loop, the OSWorld results (14.9 percent versus roughly 72 percent for humans), known failure modes, safety gating, and which workflows this technique should and should not automate today.
The Last Mile of Automation
A substantial share of enterprise work still runs through graphical interfaces that expose no API: legacy ERP clients, vendor portals, Citrix sessions, internal tools whose maintainers left years ago. Classic RPA automates these surfaces with recorded pixel positions and UI selectors. It breaks when a dialog moves, a font scales, or an unexpected popup appears. The scripts are deterministic; the screens are not. This mismatch has kept GUI automation expensive and fragile for two decades.
Large language models have been able to plan such workflows for a while. They could not execute them; they had no hands. Computer use closes exactly this gap: a model that reads the screen as pixels and returns mouse and keyboard actions in response. The relevant question is not whether this works in a demo. It is where the technique is reliable enough to deploy — and where it is not.
What Anthropic Released on October 22
On October 22, 2024, Anthropic released computer use as a public beta, alongside an upgraded Claude 3.5 Sonnet (claude-3-5-sonnet-20241022) and the announcement of Claude 3.5 Haiku. The beta is available on the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI. Anthropic's own framing is unusually blunt for a product launch: the capability is experimental and "at times cumbersome and error-prone."
The upgraded Sonnet also improves the agentic baselines that computer use depends on: SWE-bench Verified rises from 33.4 to 49.0 percent, TAU-bench retail from 62.6 to 69.2 percent — at the same price as its predecessor, 3 US dollars per million input tokens and 15 per million output tokens. Claude 3.5 Sonnet is the first frontier model to expose this capability in a public beta.
The Agent Loop: Screenshot to Action
Technically, computer use is a tool-use pattern, activated by the beta header computer-use-2024-10-22 and three Anthropic-defined tools: computer_20241022 for screenshots, mouse, and keyboard; text_editor_20241022 for file edits; bash_20241022 for shell commands. The model executes nothing itself. It returns tool calls; the client application implements every action. That client is therefore the natural enforcement point for whatever restrictions the operator wants.
The loop is plain: the client sends a screenshot, the model responds with an action and pixel coordinates, the client executes it and captures the next screenshot. Anthropic's reference implementation in the anthropic-quickstarts repository runs everything inside a Docker container with a virtual X11 display at 1024x768 and a Streamlit frontend — a disposable sandbox, not the developer's own desktop. That default is the correct one, and we recommend keeping it.
Benchmark Reality on OSWorld
The honest numbers come from OSWorld, a benchmark of 369 real computer tasks published in April 2024. The new Claude 3.5 Sonnet scores 14.9 percent in the screenshot-only category — roughly double the 7.8 percent of the next-best system — and 22.0 percent when allowed more steps per task. Trained human users reach about 72 percent. The state of the art therefore sits at one fifth of human reliability.
The failure modes are mundane. Anthropic itself names scrolling, dragging, and zooming as unreliable. Every step costs one screenshot in image tokens plus seconds of round-trip latency, and per-step errors compound over long action chains. During one recorded demo, Claude abandoned the coding task and began browsing photos of Yellowstone National Park. Anthropic reports this anecdote itself — and it captures the current reliability level precisely.
| System | OSWorld score |
|---|---|
| Claude 3.5 Sonnet (screenshot-only) | 14.9% |
| Claude 3.5 Sonnet (more steps allowed) | 22.0% |
| Next-best AI system (screenshot-only) | 7.8% |
| Human baseline | ~72% |
Safety Gating for Desktop Agents
Anthropic evaluated computer use against its Responsible Scaling Policy and kept the model at AI Safety Level 2: no new frontier risks, but concrete present-day ones. The dominant technical threat is prompt injection. An agent that reads the screen treats everything on that screen as input — including text on a prepared web page instructing it to ignore its previous instructions. No robust model-side defense exists today.
Gating therefore happens at the operator level: run the agent in an isolated VM or container, grant no access to credentials or sensitive data, require human confirmation before consequential actions, and restrict reachable domains to an allowlist. Anthropic additionally operates classifiers that flag sensitive usage patterns, with explicit reference to the upcoming US election. None of this is optional hardening. It is the precondition for running the beta at all.
What to Automate and What to Leave Alone
Our working rule at Blue IT Systems: computer use is a fallback, not a default. Where a stable API exists, the API wins — it is deterministic, faster, and orders of magnitude cheaper per operation than a screenshot loop. Screen control is justified precisely where no programmatic interface exists: legacy GUI applications, closed vendor portals, exploratory testing of user interfaces. This ordering sounds trivial; in projects it is violated regularly, because the demo impresses.
Reasonable today are supervised, read-heavy, low-blast-radius tasks in sandboxes: smoke tests of legacy applications, data extraction from GUI-only systems, form transfer between internal tools in test environments. Not reasonable are payments, credential handling, production data, irreversible actions, and unattended long-running jobs. A system at 14.9 percent on OSWorld must never be the last instance before an irreversible click. That boundary will move; today it stands here.
Outlook: Agents at the Screen
Anthropic expects the capability to improve rapidly, and we consider that plausible. Computer use is largely a perception and robustness problem, and both respond to training data and to feedback from this beta. We expect OSWorld scores to rise substantially within a year, and hybrid agents — combining accessibility trees and DOM structure with raw pixels — to become the practical norm, because pure pixel control discards information the system already has.
The more interesting shift is on the other side of the screen. If agents become regular users, operating systems and applications will grow machine-readable surfaces for them, and the screenshot becomes the fallback rather than the interface. Until then, the discipline stands: sandbox, supervision, small blast radius, APIs first. The technique is real. Production maturity is not — yet.
Sources
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (Anthropic, Oct 22, 2024)
- Developing a computer use model (Anthropic, Oct 22, 2024)
- Computer use (beta) — Anthropic API documentation (Oct 2024)
- anthropic-quickstarts: computer-use-demo reference implementation (GitHub, Oct 22, 2024)
- Xie et al.: OSWorld — Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv:2404.07972, Apr 11, 2024)