Computer use for AI agents that clicks real UI elements instead of guessed pixels. The model reads a pruned accessibility tree and says click e14. Works on macOS, Windows, Linux, and Chromium, through one MCP server.
0.4.3 captures 100x faster on macOS 26 (screencapture, per-window OCR that ignores covering apps). 0.4.2 closes the first three agent-filed issues (file dialogs by address, no whole-display OCR under an app's grant), refuses to fight the user for an app (
user_active), and asks permission through a real macOS alert. 0.4.0: the user keeps working while the agent works (docs/coexist.md).
Install
uvx a11y-computer-use doctor # try it, no install
pip install a11y-computer-use # extras: [browser] [windows] [linux]
Add it to Claude Code, Claude Desktop, Cursor, or any MCP host:
claude mcp add a11y-computer-use -- uvx a11y-computer-use mcp
{ "mcpServers": { "a11y-computer-use": { "command": "uvx", "args": ["a11y-computer-use", "mcp"] } } }
For a browser tab instead of the desktop, start Chrome with --remote-debugging-port=9222 and set A11Y_COMPUTER_USE_DRIVER=browser.
What the model sees
[snap-7] com.apple.TextEdit (window)
e1 window "Untitled"
e2 textarea ="Hello" (edit,focus)
e3 button "Save" (click)
Then it acts: click(ref="e3"), type("hello"), set_value(ref="e2", value="..."). Ref clicks go through the accessibility API and leave your pointer alone; typing addressed to an app (type(text, app=...)) leaves your screen alone too. A stale ref comes back as stale_ref with the nearest live candidates. Re-observing as a diff costs about 10 tokens. Apps with no accessibility tree (Telegram, canvases) get OCR refs o1..oN from on-device text recognition, and click(ref="o7") works the same way (docs/ocr-refs.md).
24 tools: desktop_snapshot, find, screen_text, screenshot, zoom, click, type, key, scroll, drag (with waypoint paths for strokes), wait_for, wait_until, act, set_value, scroll_to_find, notes, menu, file_dialog, app, window, clipboard, request_permission, grant_app, report_issue (agents file the tool's own defects and slow calls as issues here, docs/reporting.md), plus console, network, and webmcp (tools a page registers through navigator.modelContext, listed as w1..wN, docs/webmcp.md) on the browser. Menus and file dialogs stay reachable even in custom-drawn apps (docs/macos-primitives.md). Details: docs/agent-loop.md and the tool docstrings.
Why not the alternatives
| Approach | Native desktop | Any model | Embeddable | |
|---|---|---|---|---|
| Claude Desktop built-in | pixel loop | ✅ | ❌ | ❌ |
| browser-use, Playwright MCP | a11y tree | ❌ browser only | ✅ | ✅ |
| UI-TARS-desktop | pixel / vision | ✅ | partial | app |
| Windows-MCP, Terminator | a11y (UIA) | Windows only | ✅ | ✅ |
| a11y-computer-use | a11y tree + vision fallback | ✅ macOS, Linux, Windows*, Chromium | ✅ local too | ✅ MCP, CLI, Python |
Same planner, 13 browser tasks, one round: refs finished 13/13 with 0 misclicks for $7.04; screenshot coordinates finished 7/13 with 27 misclicks for $11.74 (full results). Run it yourself with a11y-computer-use bench h2h.
Platforms
| Observe | Ref actions | Type, keys | Coordinates, screenshot | Verified | |
|---|---|---|---|---|---|
| macOS (AX) | ✅ | ✅ | ✅ | ✅ | live on a granted Mac |
| Linux (AT-SPI2) | ✅ | ✅ | ✅ | ✅ X11, ❌ Wayland | CI + real desktop VM |
| Browser (CDP) | ✅ | ✅ | ✅ | ✅ | CI, headless Chrome |
| Windows (UIA)* | ✅ | ◐ press only | ✅ | ❌ | CI, Notepad |
*Windows is partial: ref re-resolution, capture, and coordinate input are not implemented yet. Exact gates per platform: docs/ci.md, docs/windows-port.md, docs/linux-port.md.
Safety
- Per-app grants (
read,click,full) in~/.a11y-computer-use/permissions.json. No tool can grant itself access:grant_apprecords only what the user accepted in the host's own dialog, or the user runsa11y-computer-use grant <app> <tier>. - The macOS grants come to the user: a permission error opens the system dialog and the exact settings pane and names the host app (docs/permissions.md).
- Clicks on destructive labels ask the host to confirm. No confirmation channel means the click is blocked.
- Password fields are never read, typed into, or clicked. Every action is checked against the frontmost window right before it fires.
- Everything is logged to
~/.a11y-computer-use/audit/as JSONL, with secrets redacted.
Report security issues privately: SECURITY.md.
Embed it
from a11y_computer_use import safety, server
store = safety.PermissionStore()
store.set_tier("com.apple.TextEdit", safety.Tier.FULL)
rt = server.Runtime(store=store)
print(rt.desktop_snapshot("com.apple.TextEdit", mode="interactive"))
rt.click(ref="e3", verify=True)
Existing Anthropic or OpenAI computer-use loops can run through it unchanged via a11y_computer_use.adapters (docs/provider-adapters.md). A reference agent loop ships as a11y-computer-use agent --task "..." and works with Anthropic, OpenAI-compatible endpoints (Ollama included), or the Claude Code CLI.
Spoken commands skip the planner: a11y-computer-use voice matches each utterance to one fixed skill and runs it through the same safety gate, with a per-command timeline (docs/voice.md). agent --mcp-command "ssh vm a11y-computer-use mcp" keeps the planner local and runs every action on a remote machine under its own grants.
Long jobs across several apps run as missions: phases with their own app grants, step budgets, runner-side checks (a file exists, a URL answers, text is on screen), retries, and a wall-clock timeline for video cuts.
a11y-computer-use mission run examples/missions/agency-demo.toml --provider claude-cli
Format and checks: docs/missions.md. The example mission is the design-agency reel described in docs/missions/agency-demo.md.
Docs
Agent loop · Missions · Permissions · Coexist · Reporting · Voice · WebMCP · OCR refs · macOS menus and dialogs · Adapters · Observation cost · Benchmark · Browser backend · Linux · Windows · Real-desktop test bed · CI · Decision records · Changelog · Contributing
Apache-2.0. Copyright 2026 Perception Dynamics, Inc.
1 maintainer
Metadata
1 maintainer
Release files for a11y-computer-use 0.4.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| a11y_computer_use-0.4.6.tar.gz | 705.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| a11y_computer_use-0.4.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.1 MB
Release files / a11y_computer_use-0.4.6.tar.gz
| Download URL | a11y_computer_use-0.4.6.tar.gz |
|---|---|
| Size | 705.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9b36aa121c7192c4c4ad76b342bfef2d4cd365d2247f93898272bb23ec1ffc81
|
|
BLAKE2b-256 checksum How to use checksums |
d0eef4d614045cdf6f126e21b1fffa140a8c5cde4611d05d658b9eef035c658a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency logRelease files / a11y_computer_use-0.4.6-py3-none-any.whl
| Download URL | a11y_computer_use-0.4.6-py3-none-any.whl |
|---|---|
| Size | 362.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
50971fcbb95bf44246f378ad34a88fb44fe8e92f1bee587c78511af567522e29
|
|
BLAKE2b-256 checksum How to use checksums |
065ddcd5264444bc6fd8ce1c5790f04f521ccca9582a8e42d20732dea97b37ac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency log
