a11y-computer-use 0.1.1

Computer use for AI agents that clicks real UI elements instead of guessed pixels. The model reads a pruned accessibility tree and says click e14. Works on macOS, Windows, Linux, and Chromium, through one MCP server.

Install

uvx a11y-computer-use doctor            # try it, no install
pip install a11y-computer-use           # extras: [browser] [windows] [linux]

Add it to Claude Code, Claude Desktop, Cursor, or any MCP host:

claude mcp add a11y-computer-use -- uvx a11y-computer-use mcp

{ "mcpServers": { "a11y-computer-use": { "command": "uvx", "args": ["a11y-computer-use", "mcp"] } } }

For a browser tab instead of the desktop, start Chrome with --remote-debugging-port=9222 and set A11Y_COMPUTER_USE_DRIVER=browser.

What the model sees

[snap-7] com.apple.TextEdit (window)
  e1 window "Untitled"
    e2 textarea ="Hello" (edit,focus)
    e3 button "Save" (click)

Then it acts: click(ref="e3"), type("hello"), set_value(ref="e2", value="..."). Ref clicks go through the accessibility API and leave your pointer alone. A stale ref comes back as stale_ref with the nearest live candidates. Re-observing as a diff costs about 10 tokens.

18 tools: desktop_snapshot, find, screenshot, zoom, click, type, key, scroll, drag, wait_for, act, set_value, scroll_to_find, app, window, clipboard, plus console and network on the browser. Details: docs/agent-loop.md and the tool docstrings.

Why not the alternatives

Same planner, 13 browser tasks, one round: refs finished 13/13 with 0 misclicks for $7.04; screenshot coordinates finished 7/13 with 27 misclicks for $11.74 (full results). Run it yourself with a11y-computer-use bench h2h.

Platforms

*Windows is partial: ref re-resolution, capture, and coordinate input are not implemented yet. Exact gates per platform: docs/ci.md, docs/windows-port.md, docs/linux-port.md.

Safety

• Per-app grants (read, click, full) in ~/.a11y-computer-use/permissions.json. No tool can grant itself access.
• Clicks on destructive labels ask the host to confirm. No confirmation channel means the click is blocked.
• Password fields are never read, typed into, or clicked. Every action is checked against the frontmost window right before it fires.
• Everything is logged to ~/.a11y-computer-use/audit/ as JSONL, with secrets redacted.

Report security issues privately: SECURITY.md.

Embed it

from a11y_computer_use import safety, server

store = safety.PermissionStore()
store.set_tier("com.apple.TextEdit", safety.Tier.FULL)
rt = server.Runtime(store=store)
print(rt.desktop_snapshot("com.apple.TextEdit", mode="interactive"))
rt.click(ref="e3", verify=True)

Existing Anthropic or OpenAI computer-use loops can run through it unchanged via a11y_computer_use.adapters (docs/provider-adapters.md). A reference agent loop ships as a11y-computer-use agent --task "..." and works with Anthropic, OpenAI-compatible endpoints (Ollama included), or the Claude Code CLI.

Docs

Agent loop · Adapters · Observation cost · Benchmark · Browser backend · Linux · Windows · Real-desktop test bed · CI · Decision records · Changelog · Contributing

Apache-2.0. Copyright 2026 Perception Dynamics, Inc.