a11y-computer-use 0.3.1

Computer use for AI agents that clicks real UI elements instead of guessed pixels. The model reads a pruned accessibility tree and says click e14. Works on macOS, Windows, Linux, and Chromium, through one MCP server.

0.3.1 fixes what four planner trials on a real Linux desktop found (drag waypoints, app resolution and launch, GTK and terminal content, JPEG screenshots for remote servers). 0.3.0 added voice control (a11y-computer-use voice), a remote backend (agent --mcp-command), and WebMCP tools as w refs in the browser. The full agency mission in docs/missions/ has not been run end to end yet.

Install

uvx a11y-computer-use doctor            # try it, no install
pip install a11y-computer-use           # extras: [browser] [windows] [linux]

Add it to Claude Code, Claude Desktop, Cursor, or any MCP host:

claude mcp add a11y-computer-use -- uvx a11y-computer-use mcp

{ "mcpServers": { "a11y-computer-use": { "command": "uvx", "args": ["a11y-computer-use", "mcp"] } } }

For a browser tab instead of the desktop, start Chrome with --remote-debugging-port=9222 and set A11Y_COMPUTER_USE_DRIVER=browser.

What the model sees

[snap-7] com.apple.TextEdit (window)
  e1 window "Untitled"
    e2 textarea ="Hello" (edit,focus)
    e3 button "Save" (click)

Then it acts: click(ref="e3"), type("hello"), set_value(ref="e2", value="..."). Ref clicks go through the accessibility API and leave your pointer alone. A stale ref comes back as stale_ref with the nearest live candidates. Re-observing as a diff costs about 10 tokens. Apps with no accessibility tree (Telegram, canvases) get OCR refs o1..oN from on-device text recognition, and click(ref="o7") works the same way (docs/ocr-refs.md).

21 tools: desktop_snapshot, find, screen_text, screenshot, zoom, click, type, key, scroll, drag (with waypoint paths for strokes), wait_for, wait_until, act, set_value, scroll_to_find, notes, menu, file_dialog, app, window, clipboard, plus console, network, and webmcp (tools a page registers through navigator.modelContext, listed as w1..wN, docs/webmcp.md) on the browser. Menus and file dialogs stay reachable even in custom-drawn apps (docs/macos-primitives.md). Details: docs/agent-loop.md and the tool docstrings.

Why not the alternatives

Same planner, 13 browser tasks, one round: refs finished 13/13 with 0 misclicks for $7.04; screenshot coordinates finished 7/13 with 27 misclicks for $11.74 (full results). Run it yourself with a11y-computer-use bench h2h.

Platforms

*Windows is partial: ref re-resolution, capture, and coordinate input are not implemented yet. Exact gates per platform: docs/ci.md, docs/windows-port.md, docs/linux-port.md.

Safety

• Per-app grants (read, click, full) in ~/.a11y-computer-use/permissions.json. No tool can grant itself access.
• Clicks on destructive labels ask the host to confirm. No confirmation channel means the click is blocked.
• Password fields are never read, typed into, or clicked. Every action is checked against the frontmost window right before it fires.
• Everything is logged to ~/.a11y-computer-use/audit/ as JSONL, with secrets redacted.

Report security issues privately: SECURITY.md.

Embed it

from a11y_computer_use import safety, server

store = safety.PermissionStore()
store.set_tier("com.apple.TextEdit", safety.Tier.FULL)
rt = server.Runtime(store=store)
print(rt.desktop_snapshot("com.apple.TextEdit", mode="interactive"))
rt.click(ref="e3", verify=True)

Existing Anthropic or OpenAI computer-use loops can run through it unchanged via a11y_computer_use.adapters (docs/provider-adapters.md). A reference agent loop ships as a11y-computer-use agent --task "..." and works with Anthropic, OpenAI-compatible endpoints (Ollama included), or the Claude Code CLI.

Spoken commands skip the planner: a11y-computer-use voice matches each utterance to one fixed skill and runs it through the same safety gate, with a per-command timeline (docs/voice.md). agent --mcp-command "ssh vm a11y-computer-use mcp" keeps the planner local and runs every action on a remote machine under its own grants.

Long jobs across several apps run as missions: phases with their own app grants, step budgets, runner-side checks (a file exists, a URL answers, text is on screen), retries, and a wall-clock timeline for video cuts.

a11y-computer-use mission run examples/missions/agency-demo.toml --provider claude-cli

Format and checks: docs/missions.md. The example mission is the design-agency reel described in docs/missions/agency-demo.md.

Docs

Agent loop · Missions · Voice · WebMCP · OCR refs · macOS menus and dialogs · Adapters · Observation cost · Benchmark · Browser backend · Linux · Windows · Real-desktop test bed · CI · Decision records · Changelog · Contributing

Apache-2.0. Copyright 2026 Perception Dynamics, Inc.