settled-computer

An MCP server for desktop computer use with event-driven settling: every action waits until the screen actually reacts and stops changing, then returns the settled screenshot and a one-line verdict. No fixed sleeps, no separate screenshot round trips, no model-side guessing about when to look.

Built for agent loops (Hermes, Claude Desktop, Claude Code, any MCP host). The core idea: the server answers "did the UI react, and has it stopped?" at 20 Hz with a ~1 ms frame diff, so the model doesn't need an extra vision turn just to find out when it is safe to look. Whether the reaction was the right one is still the model's job: judge by the image.

Why

Naive computer-use loops look like this:

click -> guess a sleep -> screenshot (full vision turn) -> hope

Timings are guesses: too short means acting on half-loaded UIs, too long means wasted seconds. Each "is it ready yet?" costs a model turn and a screenshot that enters context.

settled-computer replaces that with:

click -> server watches the screen -> returns the AFTER frame + verdict

• Reacted and settled in 0.4 s → Screen settled 0.40s after the action.
• Nothing changed → No visible change… (it may simply have had no visible effect: an already-focused field, an already-set state). If the model received an image within the last 45 s and the screen is unchanged, the image is omitted to save vision tokens.
• One region keeps moving (video/spinner) → bails at ~1.5 s with the region's bounding box instead of stalling 8 s, and after two consecutive bails the region is auto-ignored.

Tools

tool	what it does
act	The main one. Runs 1–8 actions in ONE call: [{"click":[x,y]}, {"type":"text"}, {"key":"enter"}]. Every step is validated before anything runs (a typo in step 5 cannot leave steps 1–4 half-executed). Each step settles before the next. A timeout always stops the sequence; "no visible change" stops it only for click/dblclick/rightclick/key (stop_on_no_reaction="auto", the default). Use "never" when a no-op is expected (e.g. the field is already focused) or "always" to stop on any no-op. The stop message lists what ran, what did not, and the exact recovery hint. screenshot="none" returns zero images on a clean run.
screenshot	Look at the screen. Always returns an image. Every action already returns the settled frame, so this is for "I have no recent frame".
click / type_text / press_key / scroll / drag / mouse_move	Single actions, each returning the settled screen + note.
wait	Wait for a slow operation to finish changing.
configure	Tune quiet_time, react_deadline, max_wait, residual_bail_after, ignore_regions. Values you set are pinned: adaptive learning and per-action defaults never override them. ignore_regions=[] also clears auto-detected regions.
screen_info	Image size (coordinates are pixels of THAT image), monitor geometry, platform, auto-ignored regions.

The server's MCP instructions field ships the usage contract to every client, including the caveat that "no visible change" is not the same as "failed".

How settling works

1. Grab a baseline frame before the action (fast reactions are never missed).
2. Perform the action (on a dedicated input thread, so a long drag or paste never blocks the loop).
3. Poll frames (burst: first poll immediately, 15 ms cadence for 100 ms, then 20 Hz), diffing full-resolution in 8×8-pixel cells (uint32 exact compare, ~1 ms per static 1080p frame).
4. Wait for: reaction (any change) → stillness (quiet_time) → return.

Adaptive max_wait (and why quiet_time is not learned)

LatencyBook learns a per-action max_wait cap (3× the p95 of past runs, floor 1 s). It records timeouts too, so an action that often times out cannot learn a short cap from its few fast runs, and it never overrides a value pinned through configure.

quiet_time is deliberately not learned. It exists to bridge pauses between UI phases (debounced search, dialog then network fetch), and the time-to-last-motion that a learner can observe says nothing about those pauses. A learned value would ratchet toward "fast but wrong" with no feedback signal.

Config precedence

configure()-pinned values › per-action defaults (type: quiet 0.12 s / react 0.2 s, scroll: quiet 0.15 s, hover: react 0.2 s) › adaptive learning (max_wait only).

Design decisions

• One lock around every tool. Hosts may issue parallel tool calls; without serialization their settle windows overlap and each verdict is contaminated by the other action.
• "No image = unchanged" has a time limit (SETTLE_MCP_ELIDE_TTL, default 45 s, 0 disables). The server cannot know what is still in the model's context (new chat on a long-lived server, hosts that prune old images, compaction), so screenshot() never omits an image and action results omit one only while the last image the model received is fresh.
• Auto-ignore with expiry. After two consecutive residual bails on overlapping regions the region is ignored for SETTLE_MCP_AUTO_IGNORE_SECS (default 30 s), then re-checked; if it is still animating it is re-activated immediately, if not it simply expires.
• A spinner is not decoration. The residual note says everything outside the region is stable and tells the model to call wait() if that region is what it is waiting for.

Safety — read this first

This server moves your mouse and types on your real desktop. It is a prompt-injection surface: any text, webpage, popup, or document visible on screen can instruct the model driving it to click and type on your behalf. Treat screen contents as untrusted input.

• Tool allowlist: SETTLE_MCP_TOOLS=screenshot,screen_info,wait strips all input injection (click/type/keys/scroll/drag/act are refused with a clear error). The single most effective hardening when you don't need full control.
• Keep the pyautogui fail-safe on (default): slam the mouse into the top-left corner to abort. The next tool call reports it as a readable error instead of a crash.
• Prefer a VM or a dedicated user session.
• act() validates every step before running and halts on timeouts and (by default) on click/key no-ops, so a mid-sequence mis-click surfaces instead of compounding.
• macOS needs Screen Recording + Accessibility permission for the host app; Linux needs X11 (Wayland blocks capture and synthetic input).

Install

pip install "settled-computer[desktop]"          # capture + input (what the MCP server needs)
pip install "settled-computer[desktop,fast]"     # + opencv: ~3x faster JPEG encode
pip install "settled-computer[desktop,unicode]"  # + non-ASCII typing (clipboard paste)

Extras: fast = opencv-python, unicode = pyperclip, desktop = mss/pyautogui/pillow. The core (just numpy + mcp) is enough to run the engine self-test or browser automation via Playwright grabbers.

Linux: pip cannot install python3-tk, but pyautogui exits without it:

sudo apt install python3-tk          # Debian/Ubuntu

Platform status (0.x alpha): Windows 10/11 is the primary, measured target (numbers below). Linux/X11 works and is Xvfb-tested in CI; Wayland blocks capture and synthetic input. macOS is unverified — the code path is expected to work with Screen Recording + Accessibility permissions granted to the host app, but no measured numbers exist yet. Hence 0.x alpha.

Works with mcp 1.x (FastMCP) and 2.x (MCPServer).

settled-computer --check     # verify capture + coordinates (console script)
settled-computer             # serve MCP over stdio
python settled_computer/engine.py --selftest   # 16 synthetic-screen tests, no display needed

Register with an MCP host

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "settled-computer": {
      "command": "settled-computer",
      "args": []
    }
  }
}

Claude Code:

claude mcp add settled-computer -- settled-computer

Hermes (config.yaml):

mcp_servers:
  settled-computer:
    command: <python-or-venv-path>
    args: ["-m", "settled_computer.server"]

Running from a source checkout instead of an install? python settle_mcp.py still works (shim into the package).

Environment variables

var	default	meaning
SETTLE_MCP_MONITOR	1	monitor index (mss numbering, 1 = primary)
SETTLE_MCP_MAX_WIDTH	1280	max width of returned images (px)
SETTLE_MCP_QUALITY	70	JPEG quality of returned images
SETTLE_MCP_FAILSAFE	1	0 disables pyautogui's fail-safe
SETTLE_MCP_ELIDE_TTL	45	seconds an unchanged screen may omit its image (0 = never omit)
SETTLE_MCP_AUTO_IGNORE_SECS	30	how long an auto-detected animating region is ignored before re-checking (0 = off)
SETTLE_MCP_TOOLS	(all)	allowlist; e.g. screenshot,screen_info,wait = observation only, input tools refused

Notes the model sees

Every action returns the settled image (unless elided, see above) plus a short note:

• Screen settled 0.38s after the action. — proceed.
• No visible change within 0.30s… It may simply have had no visible effect… — judge by the image; the click may have been fine (already focused / already set) or may have missed.
• Screen settled 1.51s after the action, except a small region [x0,y0,x1,y1] that keeps animating…
  • either an auto-ignore confirmation or the exact configure(ignore_regions=…) call.
• Screen was still changing after 8.0s (timeout). — loading; wait() then re-check. Includes the motion bounding box when motion is spread across the screen.
• Responses with no image say so explicitly and how old the last image is.
• act failures: STOPPED at step N (kind): … Done before it: … Not run: … <recovery hint>.

Known limits

• A pixel change is not proof of success: a wrong click that opens the wrong dialog also "settles".
• A blinking caret can register as a reaction, so "no visible change" is less reliable in text fields.
• Pixel-only settling cannot tell "app is thinking" from "app is done" on a static screen.
• Confined-region detection cannot distinguish a video from a progress indicator; the note says so.
• Adaptive max_wait is keyed by action type, not by application.

Performance (Windows 10, 1366×768, as measured by the author)

operation	settled-computer	native per-call driver
cheap round trip	38–55 ms (persistent stdio)	340–405 ms (process spawn)
screen capture	16 ms (mss)	387–963 ms
frame diff	1–5 ms	n/a (model compares screenshots)
3-action sequence	1 call, 1 image	≥7 model turns, ≥4 screenshots

The last row reflects act batching, not settling: any driver could batch. Run your own: python bench_mcp.py (MCP round trips) and python bench_native.py (cua-driver). python find_motion.py locates what keeps changing on your screen (e.g. to pick ignore_regions).

Safety

This server lets a model move your mouse and type on your real desktop.

• Keep the pyautogui fail-safe on (default): slam the mouse into the top-left corner to abort. The next tool call reports it as a readable error instead of a crash.
• Prefer a VM or a dedicated user session.
• act() validates every step before running and halts on timeouts and (by default) on click/key no-ops, so a mid-sequence mis-click surfaces instead of compounding.
• macOS needs Screen Recording + Accessibility permission for the host app; Linux needs X11 (Wayland blocks capture and synthetic input).

Files

settled_computer/
├── engine.py        Settle engine: wait_settled, act_and_settle, LatencyBook, selftest (no MCP dep)
└── server.py        MCP server: 11 tools, encoding, act() sequencer, tool allowlist
settle.py            shim -> settled_computer.engine (old checkouts)
settle_mcp.py        shim -> settled_computer.server (old MCP registrations)
bench_mcp.py         MCP round-trip benchmark (persistent stdio client)
bench_native.py      cua-driver benchmark (subprocess per call, for comparison)
find_motion.py       Locate perpetually-animating screen regions (ignore_regions picker)

License

MIT (see LICENSE). Chosen so MCP hosts and agent distributions can bundle it freely; any future paid tier will be an open-core split (hosted/managed features around the same open server), not a relicense.