What is this?

linux-x11-harness creates isolated X11 displays and lets AI agents control GUI applications. Each display gets its own X server, window manager, and application process group, so agents can click, type, capture screenshots, and read accessibility trees without touching the host desktop.

A live preview panel opens on your desktop while the agent works, so you can watch every click and screenshot in real time. Previews are read-only by default; double-click one to open an expanded interactive view where your mouse and keyboard directly control the sandboxed app.

Typical uses:

• Let an agent operate a GUI app without taking over your real desktop.
• Watch what the agent is doing live in a small preview window.
• Run end-to-end tests that need real input, focus, and screenshots.
• Capture accessibility trees or click UI elements programmatically.
• Discover installed/running apps and launch them by name.
• Invoke application menus by path and assert UI state after actions.
• Test GUI installers, system settings, and multi-app workflows.
• Verify accessibility compliance by auditing AT-SPI element metadata.
• Read screen text with local OCR — works even without a vision model.

See the agent skill for concrete workflows and tool usage patterns.

Quick start

Install and register with your agents in one step:

pip install linux-x11-harness
linux-x11-harness setup

After setup, open your agent and start asking it to use GUI apps. The agent will launch the harness automatically when needed.

Requirements: Python 3.10+, Rust toolchain, Linux with xvfb and openbox.

Supported agents

linux-x11-harness setup auto-configures both the MCP server and the agent skill where the agent supports them.

Other MCP-compatible agents can connect manually with linux-x11-harness mcp.

Documentation

• Usage guide — display previews, sockets, and multi-agent isolation.
• Agent skill — quick reference for agents.
• Architecture — crate layout and design notes.

License

MIT