wayland-computer-use-mcp

wayland-computer-use-mcp is a high-performance Model Context Protocol (MCP) server providing an interactive GUI testing, automation, and desktop integration suite for Wayland environments (KDE Plasma 6 / KWin, GNOME Mutter, Hyprland, Sway, and generic Wayland).

Unlike conventional "computer use" agents that rely on high-latency video streaming and expensive pixel-based coordinate guessing, wayland-computer-use-mcp implements a Tree-First Hybrid Semantic Execution Model:

• Atomic Programmatic Execution: Inspects semantic widget trees via AT-SPI2 D-Bus interfaces and executes actions (DoAction, EditableText, Text) directly without coordinate ambiguity.
• Observable Cursor Tracing: Visibly translates the pointer over target elements prior to interaction, ensuring live tracking and transparency for human observers.
• Resilient Physical Fallback: Custom canvas widgets, dropdown popovers, and complex surfaces fall back smoothly to clamped physical pointer clicks, drags, discrete wheel scrolls, and keystrokes.

⚡ Token Efficiency: 90–95% Savings Over Vision-Only Approaches

Traditional screenshot-driven computer use models stream full monitor or window screenshots on every single action, consuming 1,500 to 3,500+ vision tokens per step. A 10-step interaction sequence consumes 25,000–35,000+ tokens, introduces substantial latency, and suffers from visual coordinate hallucinations.

wayland-computer-use-mcp reduces token expenditure by over 90%:

Why This Architecture Conserves Tokens:

1. 1D Semantic Flattening: Automatically filters out invisible layout containers (GtkBox, GtkOverlay, QBoxLayout), distilling only actionable widgets into a compact array of indexed elements (b1, e1, c1) with their role, state, and label.
2. Direct Element Interaction: Programmatic invocation via interact_with_node(node_id="b1") executes atomically without requiring intermediate verification screenshots.
3. Reactive State Diffing: Every action automatically computes a pre- and post-interaction semantic delta, returning a concise markdown summary (e.g. b1 [button]: text '0 clicks' ➔ '1 clicks'), eliminating redundant tree re-polling.
4. Selective Visual Grounding: Set-of-Marks visual overlays (take_labeled_screenshot) are utilized exclusively for layout or styling validation checkpoints.

Operating Modes & Isolation Boundaries

The server operates in two distinct display modes and supports fine-grained access scopes:

1. Display Modes

• Live Mode (--live or WAYLAND_MCP_DISPLAY_MODE="live") (Default):
  • Connects to the user's active desktop session via $WAYLAND_DISPLAY.
  • Interfaces with the active session D-Bus and AT-SPI2 bus.
  • Automatically caches and reuses XDG Desktop Portal restore_token credentials to prevent repeated permission prompts.
  • Physically moves the desktop pointer so human operators can follow agent actions in real time.
• Virtual / Isolated Mode (--virtual or WAYLAND_MCP_DISPLAY_MODE="virtual"):
  • Connects to or launches an isolated virtual Wayland compositor (e.g. weston --backend=headless-backend.so, kwin_wayland --virtual, or gamescope).
  • Completely separates agent actions from personal desktop workspaces, enabling unattended, headless, or CI/CD test automation.

2. Access Scopes

• Window Isolation (--window-only or WAYLAND_MCP_ACCESS_MODE="window") (Default):
  • Prompts the user to select only the target application window in the XDG ScreenCast portal prompt.
  • Clamps all coordinate motions strictly within the detected window geometry boundaries.
• Full Display (--fullscreen or WAYLAND_MCP_ACCESS_MODE="fullscreen"):
  • Grants capture and interaction access to the entire display output.
• Dual Selection (--allow-all or WAYLAND_MCP_ACCESS_MODE="both"):
  • Allows either window or monitor selection during the portal handshake.

🔒 Security Architecture & Upstream Portal Confinement Disclosure

How wayland-computer-use-mcp Mitigates This Risk:

To guarantee safe operation despite upstream protocol constraints, this server implements five layers of client-side containment:

1. Strict Coordinate Boundary Clamping (CoordinateClamper): All injected pointer coordinates are mathematically clamped to $[0 \le x \le W, 0 \le y \le H]$ of the target application surface. The server strictly forbids emitting coordinates outside the active window frame.
2. Dynamic Geometry Drift Detection (GeometryDivergenceDetector): Monitors the baseline window surface position and dimensions. If a window moves, resizes, or unminimizes unexpectedly while an action is pending, the operation is immediately aborted to prevent clicks from spilling into adjacent desktop surfaces.
3. Hardware User Preemption (UserInterventionDetector): Tracks physical hardware cursor activity. If the user moves the physical mouse or types on the keyboard, automated interactions pause instantly to yield control to the human operator.
4. Dangerous Shortcut Blacklist (ShortcutFilter): Blocks hazardous keyboard sequences (e.g. Super/Meta, Ctrl+Alt+Delete, Alt+F4, VT terminal switching).
5. Virtual Display Sandbox Recommendation: For evaluating autonomous agents or untrusted scripts, execute with --virtual to provide hardware-level process and display server isolation.

Quickstart

Run Directly via uvx (Zero installation required)

uvx wayland-computer-use-mcp

Linux Distribution Prerequisites

wayland-computer-use-mcp utilizes native Wayland portals, AT-SPI2 accessibility D-Bus, and PyGObject for GTK4/Adwaita automation:

• Debian / Ubuntu / Pop!_OS:

  sudo apt install -y python3-gi python3-gi-cairo gir1.2-gtk-4.0 gir1.2-adw-1 at-spi2-core dbus-x11 wl-clipboard
  

• Fedora / RHEL:

  sudo dnf install -y python3-gobject gtk4 libadwaita at-spi2-core dbus-x11 wl-clipboard
  

• Arch Linux / Manjaro:

  sudo pacman -S --needed python-gobject gtk4 libadwaita at-spi2-core dbus wl-clipboard
  

Install in Virtual Environment

git clone https://github.com/Niklas-Ehrenfried/wayland_computer_use_mcp.git
cd wayland_computer_use_mcp
uv venv --python python3 --system-site-packages
source .venv/bin/activate
uv pip install -e .

MCP Client Configuration

1. Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

2. Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp", "--live", "--window-only"]
    }
  }
}

3. Antigravity IDE / Gemini Code Assist (mcp_config.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

4. Roo Code / Cline (cline_mcp_settings.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

Exposed Tool Suite (25 Tools)

1. Tree-First Semantic Navigation & Macro Presets (5 Tools)

• inspect_ui_tree(pid, max_depth): Returns the collapsed 1D interactive element list (b1, e1, c1, sw1, li1) with widget names, roles, states, and coordinates.
• interact_with_node(node_id, action, text, settle_timeout_ms, ...): Dispatches semantic interactions. Visibly traces cursor, executes AT-SPI action, waits for D-Bus UI event settlement, and falls back to physical input if required. Returns structured JSON UI deltas.
• batch_actions(actions, pid): Executes a batch of sequential UI operations atomically with step-by-step delta tracking without intermediate screenshot pauses.
• preset_workflow(action, name, description, steps, pid): Manages and executes reusable multi-step interaction presets (list, run, view, save, delete) stored as persistent JSON artifacts under .agents/artifacts/presets/.
• watch_ui_events(pid, timeout_seconds): Streams AT-SPI2 D-Bus accessibility events (Object:StateChanged, ChildrenChanged, TextChanged, Window:Activate) directly to observe async UI changes.

2. Clamped Physical Input (8 Tools)

• click(x, y, button, pid): Executes a mouse click clamped to window bounds.
• double_click(x, y, button, pid): Dispatches a standard mouse double-click.
• right_click(x, y, pid): Dispatches a right-click (context menu).
• hover(x, y, duration_ms, pid): Moves pointer without clicking, activating Wayland tooltips or hover highlights.
• drag(start_x, start_y, end_x, end_y, pid): Performs a clamped mouse drag gesture.
• scroll(dx, dy, pid): Dispatches pointer wheel ticks via NotifyPointerAxisDiscrete and continuous deltas.
• type_text(text, x, y, pid): Types text using evdev keycodes with automated clipboard paste fallback for strings > 30 characters.
• key_combination(keys, pid): Sends modifier hotkeys (e.g. ["ctrl", "s"], ["alt", "tab"]). All 8 input tools accept an optional pid to explicitly target a specific application window.

3. Visual Grounding & Inspection (2 Tools)

• capture_window_frame(crop_box, save_artifact): Captures a high-resolution window frame. In-memory MCP ImageContent by default; saves rolling disk cache when save_artifact=True.
• take_labeled_screenshot(save_artifact): Captures window frame annotated with Set-of-Marks boundary badges rendering compact semantic IDs ([b1], [e1], [sw1]) for zero-friction mental mapping to interact_with_node, covering all interactive widgets (including switches, list items, tree items, and table cells).

4. Process Lifecycle & Crash Interception (5 Tools)

• launch_app(target, args, restart, cwd): Spawns Python GUI scripts or system apps with automatic virtual environment discovery. Immediately returns the initial interactive element tree so agents can act without a separate inspect_ui_tree call.
• restart_app(pid): Gracefully terminates and re-launches an active process, preserving original launch arguments and working directory. Strictly validates process ownership: unmanaged or user-opened applications raise PermissionError to preserve desktop consistency.
• terminate_app(pid): Terminates application processes cleanly (SIGTERM escalated to SIGKILL).
• list_managed_apps(): Lists all active processes managed by the MCP server, pruning dead PIDs.
• get_app_logs(pid, lines): Retrieves console output and crash tracebacks from a thread-safe circular buffer.

5. OS & Desktop Integration (5 Tools)

• clipboard_read(): Reads text from the Wayland clipboard (wl-paste).
• clipboard_write(text): Writes text to the Wayland clipboard (wl-copy).
• window_control(action, pid): Controls window state (minimize, maximize, restore, close).
• install_to_desktop(app_id, name, ...): Generates a valid Linux .desktop launcher with worktree detection and version badging.
• uninstall_from_desktop(app_id): Removes desktop launchers and associated icons.

MCP Prompts & Resources

• Prompt: wayland_automation_guide: Best-practice agent instructions for tree-first navigation, delta-to-delta flow, and fallback rules.
• Resource: wayland://system_prompt: Dynamic guidelines for LLM agent integration into client system prompts.

Interactive Test Rig

A complete 14-component GTK4/Adwaita verification application is provided in examples/test_gui_app.py:

uv run python examples/test_gui_app.py

Run the automated live end-to-end integration test suite:

uv run pytest tests/test_live_example_app.py -v

License

MIT License. See LICENSE for details.