hermes-computer-use

Scope: Windows 11 + WSL2 Ubuntu 22.04 / 24.04 only. This project intentionally limits its support matrix — native Linux / macOS / Windows are not targets. See docs/WSL_SETUP.md for why and for the full setup walkthrough.

Pixel-level browser automation MCP server. Gives any MCP-speaking agent (hermes-agent, Claude Code, Codex, …) 21 tools to drive a real Chrome browser running in an Xvfb display: screenshots as vision input, OS-level mouse/keyboard as output. No CDP. No navigator.webdriver. No DOM shortcuts.

Think of it as the Linux-side reproduction of Anthropic's computer-use-demo — but exposed over stdio MCP so you can pair it with any agent runtime and any vision-capable model.

agent ── stdio MCP ──▶ hermes_computer_use.server ── subprocess ──▶ xdotool / scrot
                                                                          │
                                                                          ▼
                                                                      Xvfb :99
                                                                          │
                                                          ┌───────────────┴────────────────┐
                                                          ▼                                ▼
                                                    x11vnc :5900              websockify + noVNC :6080
                                                (native VNC clients)            (browser viewer)

See docs/ARCHITECTURE.md for the longer version.

Why

If your automation has to walk a login funnel on a site with Cloudflare, Kasada, or reCAPTCHA sprinkled on it, this stack usually passes where Playwright gets stopped — because the browser is indistinguishable from a stock Chrome driven by a stock X server.

Install

Prerequisites (Windows host): Windows 11, WSL2 with an Ubuntu 22.04 or 24.04 distro, and systemd enabled in WSL. Full walkthrough in docs/WSL_SETUP.md.

Everything below runs inside the WSL shell, not in PowerShell.

git clone https://github.com/Noah3521/hermes-computer-use.git ~/hermes-computer-use
cd ~/hermes-computer-use

# 1. System packages (sudo): Xvfb, fluxbox, x11vnc, xdotool, ydotool, scrot,
#    ImageMagick, CJK fonts, Google Chrome, plus uinput if available.
bash scripts/setup.sh

# 2. Python package
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[novnc]"       # omit [novnc] if you don't want the web viewer

# 3. Optional browser-based observer at http://localhost:6080/vnc.html
bash scripts/install-novnc.sh

# 4. Persistent services
mkdir -p ~/.config/systemd/user
cp systemd/computer-use.service.example ~/.config/systemd/user/computer-use.service
cp systemd/novnc.service.example        ~/.config/systemd/user/novnc.service
sudo loginctl enable-linger "$USER"
systemctl --user daemon-reload
systemctl --user enable --now computer-use.service novnc.service

Smoke test:

python examples/smoke_test.py

Wire to hermes-agent

Paste config/hermes.yaml.example into your ~/.hermes/config.yaml under mcp_servers:, then hermes gateway run --replace. The model immediately gets the full tool surface.

The same config shape works for any stdio-MCP client (Claude Code, mcp-inspector, custom runners).

Tools

Full signatures live in src/hermes_computer_use/server.py and are discoverable via MCP tools/list.

Demo prompts

examples/demo_prompts.md ships ten graduated prompts from a 5-second sanity check to a 5-hop Google → external site → SSO-login flow that passes without captchas. Open the noVNC tab while running them — watching the pointer interpolate through Google's search box is surprisingly compelling.

Configuration

All runtime behaviour is controlled by env vars. Sensible defaults everywhere.

Troubleshooting

See docs/TROUBLESHOOTING.md. The usual suspects:

• scrot: Can't open X display → Xvfb died. systemctl --user restart computer-use.service.
• Chrome immediately exits → sandbox / dev-shm issue. The scripts/display.sh launcher already sets the right flags; if you hand-roll, copy from there.
• Stack dies on logout → sudo loginctl enable-linger $USER.
• Google flags "unusual traffic" → IP reputation, not behavioural. Use a residential proxy or prewarm with a manual login via VNC.

Security

This is an LLM with hands. Read docs/SECURITY.md before pointing it at anything you care about. At minimum:

• Run in an isolated WSL distro or VM — never your daily driver.
• Remove the run_shell tool if the agent does not need a shell.
• Do not persist real credentials in CU_PROFILE_DIR.

Contributing

See CONTRIBUTING.md. Scope guardrails are strict: no DOM selectors, no OCR, no anti-detection arms race. The thesis is "emit no abnormal signals" > "emit clever evasions".

License

MIT. See LICENSE.

Acknowledgements

• anthropic-quickstarts/computer-use-demo for the reference loop.
• x11vnc + noVNC for the observer pipeline.
• Model Context Protocol for making "tool surface you can point any agent at" a real thing.