computer-use-mcp ("realhands")

An MCP server that lets Claude operate your real computer the way a human does — moving the actual mouse, clicking, typing, and reading the actual screen.

Unlike OpenAI Operator, browser-use, or Playwright agents (which spin up a separate, isolated, logged-out Chrome), this drives the physical OS cursor and keyboard. So it works in your own Chrome with your own logged-in sessions — and in every other app — because it's just a human at the keyboard, as far as any website can tell.

Status: LIVE and battle-tested. Registered with Claude Code as the user-scope MCP realhands (tool mcp__realhands__computer) and ✓Connected since 2026-06-02. On 2026-06-09 it drove the user's real, logged-in Chrome through a complete Google Play Console deployment (app upload, release notes, submission) end-to-end.

The server is registered as realhands rather than computer-use because the name "computer-use" is reserved in Claude Code.

How it works

Claude (Desktop or Code) is the agent loop. You type a task; Claude calls the single computer tool in a see → think → act cycle:

See — screenshot returns the real screen (downscaled to ~1280px for grounding accuracy) → Think — Claude picks the next action + pixel coordinates → Act — the server moves the real mouse / types on the real keyboard → a fresh screenshot comes back automatically after every action, and it repeats.

Two Windows-specific details make clicks land accurately (src/screen.py):

• DPI awareness — SetProcessDpiAwareness(2) is set at import time so screenshot pixels == pyautogui cursor coordinates even under display scaling (125% / 150% / …).
• Stateless coordinate scaling — screenshots are downscaled (LANCZOS) to at most COMPUTER_USE_MAX_DIM on the longest side before sending; incoming click coordinates are scaled back up to real pixels. The scale factor is a pure function of monitor geometry + MAX_DIM, so mapping never depends on which screenshot ran last. Coordinates are clamped inside the target monitor so a stray click can't fly off-screen. Each axis is rounded down to a whole 28px vision patch, so the two axes can scale very slightly differently (<2.2%); to_real() maps each with its own factor and stays exact.

Multi-monitor: every call takes an optional monitor index (1 = primary, 2.. = others, 0 = the whole virtual desktop). action="monitors" enumerates the setup. Origins may be negative for screens left/above the primary — to_real() handles the offset. Use the same monitor for a click as for the screenshot you're clicking on.

Architecture

src/realhands/
  server.py   FastMCP server "computer-use"; the single `computer` tool (action enum
              modeled on Anthropic's reference computer_20250124 tool); runs one
              action or a batch of `steps`, then returns status text + at most one
              screenshot
  screen.py   DPI awareness, mss capture, patch-aligned downscale, model-space ->
              real-pixel mapping, unchanged-screen detection
  input.py    pyautogui mouse/keyboard execution; xdotool-style key-name translation
              (Return, Page_Down, ctrl+a, super, ...); clipboard-paste fast path for
              long/Unicode/multiline typing (preserves your existing clipboard);
              activate_window via win32 AttachThreadInput
  safety.py   kill switches + lazy arm / stand-down lifecycle
  config.py   .env-driven configuration (all defaults are sensible; .env is optional)
install.py    one-shot installer: venv, deps, .env, Claude Desktop registration

Stack: Python 3.10/3.11 · mcp (FastMCP, stdio) · pyautogui · mss · pillow · pynput · keyboard · pyperclip · python-dotenv — plus pygetwindow and pywin32 for activate_window.

The computer tool

A single tool with an action parameter:

Coordinates are in the pixel space of the most recent screenshot; its size is reported with every capture. After every non-screenshot action the tool waits ~0.4s for the UI to settle and returns a fresh screenshot.

Token economy

Screenshots dominate the cost of driving a desktop, and not just once: every image stays in the conversation and is re-sent as history on each later turn. Claude bills vision in 28×28 patches — tokens = ⌈w/28⌉ × ⌈h/28⌉ — so this server does four things to keep the bill down.

Batch steps. Pass steps (a list of action dicts) instead of one call per action and the whole run shares one screenshot at the end:

{"steps": [{"action": "left_click", "coordinate": [420, 300]},
           {"action": "type",  "text": "hello@example.com"},
           {"action": "key",   "text": "Tab"},
           {"action": "type",  "text": "secret"},
           {"action": "key",   "text": "Return"}]}

That is 1 125 visual tokens instead of 5 625, and one round trip instead of five — the larger saving, since each avoided turn also avoids re-sending the entire transcript. A failing step stops the run, reports which step failed, and still returns the screen. Add "screenshot": false to skip the trailing image too.

Patch-aligned downscaling. A dimension that isn't a multiple of 28 pays for a partial patch row/column carrying almost no pixels. From a 1920×1080 primary the default 1260×700 is exactly 45×25 patches = 1 125 tokens, versus 1 196 for 1280×720 — 6% off for 1.5% fewer pixels.

Unchanged-screen suppression. If under CHANGE_THRESHOLD of pixels moved since the last image sent, the reply is a line of text instead of a screenshot. A real desktop never produces two byte-identical frames (clock, caret, hover states), so this is a threshold, not an equality check. After MAX_SKIPS suppressions in a row it force-sends one, so the model can't fly blind if it lost the earlier image to context compaction.

Right-sized images. COMPUTER_USE_MAX_DIM trades grounding accuracy against cost — from a 1920×1080 primary: 1792 → 2 304 tokens, 1260 → 1 125 (default), 1036 → 777, 896 → 576. Keep the long edge ≤ 2576 px: an image returned inside a tool_result is rejected rather than downscaled when it exceeds the model's limit.

COMPUTER_USE_IMAGE_FORMAT is not a token lever — Claude bills by pixel dimensions, so a JPEG and a PNG of the same screenshot cost exactly the same. JPEG only cuts payload bytes (~977 KB → ~141 KB here), which helps latency at some risk to small-text legibility.

Safety — it controls your REAL machine

This is fully autonomous: it does not ask before each action. Three independent kill switches (src/safety.py):

1. Fail-safe corner — slam the mouse into the top-left corner → pyautogui raises FailSafeException and the action aborts instantly.
2. Panic hotkey — Ctrl+Alt+Q (configurable) → hard-kills the server process (os._exit(1)).
3. STOP overlay — an always-on-top window (top-right) showing the current action, with a big red ■ STOP AGENT button that also hard-kills the process.

Lazy arm / stand-down: the overlay and the global panic hotkey are armed lazily on the first action of a task, not at server startup — idle sessions show nothing and grab no hotkeys. They stand down when the agent calls action="stop" at the end of a task, and re-arm automatically on the next action. (The STOP overlay is a single persistent window that is hidden when dormant, never destroyed — recreating it was a crash hazard.) An optional idle auto-stand-down is available via COMPUTER_USE_IDLE_STOP but is disabled by default: an agent's thinking time between tool calls easily exceeds any short idle window, so a non-zero value would stand the agent down mid-task.

Pacing also helps you stay in control: every action is followed by a configurable pause (COMPUTER_USE_PAUSE) and the cursor glides rather than teleports (COMPUTER_USE_MOVE_DURATION), so you can watch and interrupt.

Don't leave it unsupervised on anything that can spend money, send messages, or delete data.

Install

Requires Python 3.10 or 3.11 (3.13+ untested; avoid the 3.14 beta).

Recommended: uvx (always the latest version)

Nothing to install up front, and you get every release automatically — uvx resolves the newest published version each time the server starts, so a restart of your MCP client is the whole upgrade process. Requires uv.

uvx realhands@latest

Drop the @latest (uvx realhands) if you would rather let uv reuse whatever version it already has cached.

From PyPI (pinned)

pip install realhands

This installs the realhands console script and the importable realhands package. Run the server with either realhands or python -m realhands.server.

Pick this over uvx when you want a pinned version that never changes underneath you, or when the machine may be offline at startup — uvx realhands@latest needs to reach PyPI each time it launches. The trade-off is that upgrades become manual:

pip install --upgrade realhands

Then restart your MCP client — clients bind their servers at session start, so a running session keeps the old code until it restarts.

From source (with Claude Desktop registration)

git clone https://github.com/kanishka089/computer-use-mcp
cd computer-use-mcp
py -3.10 install.py

This creates .venv/, installs the package + deps (editable), copies .env.example to .env if missing, and registers the server in Claude Desktop's config (backing up any existing config). Restart Claude Desktop, then look for the computer-use tool.

Claude Code

Register it as a user-scope stdio server named realhands. This form auto-upgrades — each new session resolves the latest release, so you never have to think about it:

claude mcp add realhands --scope user -- uvx realhands@latest

The tool then appears as mcp__realhands__computer in every project.

If you installed with pip instead and want a pinned version, point it at the Python that has the package (and run pip install --upgrade realhands yourself to move up):

claude mcp add realhands --scope user -- python -m realhands.server

Already registered on an older version? Re-point it at the auto-upgrading form:

claude mcp remove realhands --scope user
claude mcp add realhands --scope user -- uvx realhands@latest

Removing and re-adding an MCP mid-session drops this session's connection to it — the new registration is picked up on the next Claude Code start.

Use

Just ask. For example:

Take a screenshot, open Chrome, go to YouTube, and search for "lofi".

Watch your real cursor move and your logged-in Chrome respond. Real-world proof: it has autonomously completed a full Google Play Console release flow in the user's own signed-in Chrome session.

Configuration (.env, optional — defaults are fine)

Known gotchas

• MCP connection drops when the agent idles between turns. The stdio connection to realhands can silently die while Claude is thinking/waiting between turns. Fix: issue a screenshot action — it silently reconnects. Importantly, an action that "failed" with Connection closed often still executed on the real machine — take a screenshot and check the actual screen state before retrying, or you may double-click / double-submit.
• Click coordinates must match the screenshot's monitor. If you screenshot monitor=2 and then click without passing monitor=2, the click lands on the primary.
• activate_window beats the taskbar. Windows' foreground-lock makes taskbar clicks unreliable (the icon just flashes). activate_window uses AttachThreadInput + z-order toggling + a minimize/restore fallback, so prefer it for app switching.
• Don't run with Python 3.13/3.14. Tested on 3.10/3.11 only; the installer warns.
• Typing long/Unicode text uses the clipboard. Your clipboard is saved and restored, but anything watching the clipboard will see the pasted text momentarily.

Self-test

python -m realhands.screen

Captures the screen, prints real vs. sent dimensions and the scale factor, writes test_capture.png, and runs a coordinate round-trip check (center + both corners).