---

Overview

Wayland MCP Server lets AI assistants interact with a Wayland desktop through the Model Context Protocol. It captures the screen for visual analysis by a vision language model, controls the pointer and keyboard, and chains actions into multi-step workflows.

It was built around two observations. Screenshot and automation tooling on Wayland is fragile: it depends on which binaries are installed, which protocol extensions the compositor advertises, and which portals are present on the session bus. And the usual remedy, giving an automation tool root or write access to /dev/input, is a security hole the user pays for every day.

This server answers both. It probes the actual capabilities of the session and picks the best backend that can work, and its preferred path needs no privilege at all: the compositor itself mediates input through the XDG portals.

Quick example

# AI Assistant: "Take a screenshot and tell me what's on screen"
#   Captures the screen, analyzes it with a VLM, and describes what it sees.

# AI Assistant: "Click the OK button"
#   Locates the button from the screenshot, moves the pointer, and clicks.

# AI Assistant: "Fill out this form with test data"
#   Chains pointer and keyboard actions to complete the form.

Features

Visual analysis

• Screenshot capture with precision ruler overlays
• VLM-powered image analysis via OpenRouter or Google Gemini
• Multiple vision models (Claude, GPT-4V, Gemini, Qwen)
• Side-by-side image comparison and diff detection

Pointer automation

• Absolute and relative pointer positioning
• Click operations (left, right, middle)
• Drag and drop with coordinate precision
• Bidirectional scrolling (vertical and horizontal)

Keyboard control

• Text input simulation
• Individual key press events
• Complex key combinations

Action sequences

• Chain multiple operations together
• Flexible syntax: chain:action1;action2;action3
• Example: chain:click:100,200;type:hello;press:Enter

Installation

Prerequisites

• Python 3.10 or higher
• A Wayland session (GNOME, KDE Plasma, COSMIC, Hyprland, Sway, ...)
• No privileged setup and no root. See Backends for what each path needs; the server reports it itself through the describe_environment tool.

PyGObject is recommended but optional: it unlocks the XDG portal backends, the only path that works on every desktop.

sudo apt install python3-gi        # Debian, Ubuntu, Pop!_OS
sudo dnf install python3-gobject   # Fedora
sudo pacman -S python-gobject      # Arch

The portal capture backend works without GStreamer. These packages are only needed for the cursor in portal screenshots: GStreamer's PipeWire source and its introspection data, e.g. sudo apt install gstreamer1.0-pipewire gir1.2-gstreamer-1.0 on Debian and Ubuntu.

Quick install

uvx wayland-mcp-kurojs

With pip, into any environment:

pip install wayland-mcp-kurojs

To run the very latest main instead of the released build, install from this repository:

uvx "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"

The wayland-mcp name on PyPI belongs to an older upstream build; this project publishes as wayland-mcp-kurojs. To let the ephemeral environment see a system PyGObject, use a virtual environment that keeps system packages visible:

uv venv --system-site-packages
uv pip install "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"

From source

git clone https://github.com/kurojs/wayland-mcp.git
cd wayland-mcp
uv venv --system-site-packages
uv pip install -e ".[dev]"

Input control setup

There is none. Input goes through the org.freedesktop.portal.RemoteDesktop portal, which synthesizes pointer and keyboard events with no privilege at all. Your compositor asks for permission the first time; the returned restore token is kept in $XDG_STATE_HOME/wayland-mcp/remote-desktop-token (mode 0600) and replayed afterwards, so you are not asked again. Set WAYLAND_MCP_NO_PERSIST=1 to be prompted every time instead.

If you ran the upstream sudo ./setup.sh, undo it. That script made every /dev/input/event* device world-writable and installed a udev rule (KERNEL=="event*", MODE="0666") to keep them that way across reboots, plus a setuid bit and a NOPASSWD sudoers entry for evemu-event. While that rule is in place, any local process can read your keystrokes, passwords included. scripts/legacy-evemu-setup.sh documents the exact rollback commands. This project removed the script from the install path and needs none of it.

Usage

MCP configuration

The server supports two VLM providers.

Option 1: OpenRouter (multiple models via proxy)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "OPENROUTER_API_KEY": "sk-or-v1-...",
        "VLM_PROVIDER": "openrouter",
        "VLM_MODEL": "qwen/qwen2.5-vl-72b-instruct:free",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Option 2: Google Gemini direct (native API, faster)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Example for Claude Desktop (~/.config/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Environment variables

Everything below is optional. The server starts, captures the screen and controls input with none of it set.

Variable	Description	Default
Vision analysis (optional feature)
VLM_PROVIDER	openrouter, gemini or azure	openrouter
OPENROUTER_API_KEY	OpenRouter key	-
GEMINI_API_KEY	Google Gemini key	-
AZURE_API_KEY, AZURE_ENDPOINT, AZURE_DEPLOYMENT	Azure AI Foundry	-
VLM_MODEL	Model identifier	provider default
WAYLAND_MCP_CONFIG	JSON file to read a key from, if you keep it out of the environment	-
Backend selection
WAYLAND_MCP_CAPTURE_BACKEND	Force a capture backend by name	auto
WAYLAND_MCP_INPUT_BACKEND	Force an input backend by name	auto
WAYLAND_MCP_NO_PERSIST	1 to be asked for portal permission every time	unset
WAYLAND_MCP_CURSOR_STREAM	1 to try the ScreenCast stream for the cursor without a stored permission, which may prompt; 0 to never use it	unset: only with a stored permission
WAYLAND_MCP_RESTORE_TOKEN_PATH	Where to keep the portal restore token	$XDG_STATE_HOME/wayland-mcp/remote-desktop-token
Behaviour
WAYLAND_MCP_QUIET_CAPTURE	1 to silence animations and sound around a capture, restoring your settings afterwards	unset
WAYLAND_MCP_LOG	Log file path	/tmp/wayland-mcp.log
Session
XDG_RUNTIME_DIR, WAYLAND_DISPLAY, DBUS_SESSION_BUS_ADDRESS	Recovered automatically when your MCP client strips them; set them explicitly if you run several compositors	auto

Getting API keys:

• OpenRouter: openrouter.ai → Keys section
• Google Gemini: Google AI Studio

MCP clients strip the environment. The reference stdio transport passes only HOME, LOGNAME, PATH, SHELL, TERM and USER to the server it launches, which removes exactly what a Wayland tool needs. The server reconstructs XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and WAYLAND_DISPLAY from the filesystem when they are missing, so no client-side configuration is required. It declines to guess only when several compositor sockets are present; set WAYLAND_DISPLAY in your client config then.

Backends

Backends are chosen by capability, never by compositor name: the server probes which binaries exist and at what version, which portal interfaces are on the session bus, and which globals the compositor advertises, then picks the highest priority backend whose requirements are actually met. Call describe_environment to see what it decided and why.

Capture, highest priority first:

Backend	Needs	Notes
cosmic-screenshot	the binary, shipped with COSMIC	silent, no dialog; full screen only
grim	grim, plus zwlr_screencopy_manager_v1, or ext_image_copy_capture_manager_v1 with grim >= 1.5	region capture via slurp
ksnip, gnome-screenshot, spectacle	the respective binary	the upstream cascade, preserved
portal	org.freedesktop.portal.Screenshot + PyGObject	works everywhere; may prompt. Draws the cursor from a ScreenCast stream when a portal permission is already stored, and the cursor field of the result says which path produced the image (see below)

Input, highest priority first:

Backend	Needs	Pointer	Keyboard
portal	org.freedesktop.portal.RemoteDesktop + PyGObject	yes, absolute	yes, layout-independent
wtype	wtype + zwp_virtual_keyboard_manager_v1	no	yes
ydotool	ydotool + writable /dev/uinput	yes	yes, US layout only
evemu	evemu-event + an already writable /dev/input/event*	approximate	yes, US layout only

evemu is last on purpose and is never a requirement. Nothing in this project changes /dev/input permissions.

Desktop environment compatibility

Desktop	Capture	Input	Notes
COSMIC (Pop!_OS 24.04)	Yes (cosmic-screenshot)	Yes (portal)	Verified end to end on cosmic-comp 0.1. Only implements ext-image-copy-capture, so a grim older than 1.5 cannot capture here.
GNOME	Yes (gnome-screenshot or portal)	Yes (portal)	No zwp_virtual_keyboard, so wtype is unavailable; the portal covers it.
KDE Plasma	Yes (spectacle or portal)	Yes (portal)
Hyprland, Sway, wlroots	Yes (grim)	Partial: wtype for keyboard; pointer needs ydotool	xdg-desktop-portal-wlr provides ScreenCast and Screenshot but not RemoteDesktop, so there is no portal pointer path.
Others	Partial: falls back to portal	Depends	describe_environment will tell you.

Only the COSMIC row was verified on real hardware; the others follow from each backend's stated requirements and the unit tests that pin the selection, not from a live run.

Known limitations

• Region capture needs grim + slurp, or a portal that offers an interactive picker. cosmic-screenshot cannot select a region without a human, so it reports that instead of blocking an MCP call.
• Keyboard layout: the portal backend types by keysym and is layout independent. ydotool and evemu type by position, so on a non-US layout they produce whatever those positions mean there. Asking for ASAP 42 on AZERTY gives QSQP 'é.
• cosmic-comp discards the first synthesized event after a session starts. The portal backend absorbs that with a zero-distance warm-up motion; without it, the first click or keystroke of every session is silently lost.
• Absolute pointer positioning through the portal needs a ScreenCast stream attached to the session. Where the portal refuses one, absolute moves raise an explanatory error instead of silently misplacing the cursor; use a relative move, or another input backend.
• The cursor in portal screenshots: the Screenshot portal has no cursor option, and GNOME leaves the pointer out. With include_mouse (the default) the portal capture backend therefore takes the frame from a RemoteDesktop session's ScreenCast stream, which embeds the cursor. It needs GStreamer's pipewiresrc, and by default a portal permission has to be stored already -- using any input tool once stores one -- because opening that session without one is what raises the remote-control dialog, and a screenshot is not worth that. WAYLAND_MCP_CURSOR_STREAM=1 forces the attempt without a stored permission and may prompt; =0 turns the cursor path off. A stream that fails falls back to the Screenshot portal for the following 60 seconds, and is tried again after that. A locked screen, where GNOME refuses new sessions, only costs the cursor of that one capture. The cursor field of the result says which one you got.

Example commands

Through an MCP client, you can request actions like:

• "Take a screenshot and analyze what's on the screen"
• "Move the mouse to coordinates (100, 200) and click"
• "Type 'hello world' and press Enter"
• "Click at (50, 50), then drag to (200, 200)"

Available tools

The server exposes the following MCP tools.

Screen capture

• capture_screenshot - take a screenshot with optional ruler overlays
• capture_and_analyze - capture and analyze using VLM in one step

Vision analysis

• analyze_screenshot - analyze an existing screenshot with custom prompt
• compare_images - compare two screenshots to detect differences

Pointer control

• move_mouse - move cursor to coordinates (absolute or relative)
• click_mouse - perform left click at current position
• drag_mouse - drag between two coordinate points
• scroll_mouse - vertical scroll (positive=up, negative=down)

Action execution

• execute_action - execute a single action or chain multiple actions

Action chain syntax

Combine multiple actions with semicolons:

chain:action1;action2;action3

Supported actions:

• type:text - type a text string
• press:key - press a specific key
• click: or click:x,y - click at position or current location
• move_to:x,y - move to absolute coordinates
• move_to:rel:x,y - move relative to current position
• drag:x1,y1:x2,y2 - drag from point to point
• scroll:amount - scroll vertically (typical values: 15-120)
• scroll:horizontal:amount - scroll horizontally

Example chains:

chain:move_to:100,200;click:;type:hello;press:Enter
chain:click:50,50;drag:50,50:200,200
chain:scroll:120;move_to:rel:0,-50;click:

Security

This server grants extensive control over your desktop environment: full pointer and keyboard control, screen capture, and the ability to execute arbitrary input sequences. Only use it with trusted AI models and MCP clients, review action chains before execution in sensitive contexts, and consider running it in a sandboxed or test environment. The AI can perform any action you could perform manually.

Permission model

No setuid binaries, no sudoers entries, no udev rules, no input group membership. The nominal path is the XDG portal: your compositor asks you once, mediates every event, and can revoke the grant at any time. The stored restore token is a capability. Treat it like a credential. Delete $XDG_STATE_HOME/wayland-mcp/remote-desktop-token to revoke it locally, and revoke the permission in your desktop settings to invalidate it entirely.

The fallback backends need more, and get chosen only when nothing better exists: ydotool wants a writable /dev/uinput, evemu a writable /dev/input/event*. The server checks whether that access already exists. It never asks for it, and it never widens it.

Architecture

                    +-----------------------------------+
                    |         MCP Client Layer          |
                    |    (Claude, Cursor, VS Code)      |
                    +------------------+----------------+
                                       |
                              MCP Protocol (stdio/HTTP)
                                       |
                    +------------------v----------------+
                    |        Wayland MCP Server         |
                    |    +----------------------+       |
                    |    |    Core Components    |       |
                    |    +----------------------+       |
                    |    |   FastMCP Handler     |       |
                    |    |   Action Processor    |       |
                    |    |   Chain Parser        |       |
                    |    +----------------------+       |
                    +----+-----------------+------------+
                         |                 |
        +----------------v----+      +-----v-------------+
        |      Vision         |      |       Input       |
        |      Control        |      |      Control      |
        +---------------------+      +-------------------+
        |  VLM analysis       |      |  portal           |
        |  image comparison   |      |  wtype            |
        |                     |      |  ydotool          |
        +---------------------+      +-------------------+
                         |                 |
                         +--------+--------+
                                  |
                        +---------v---------+
                        |  Screen Capture   |
                        +-------------------+
                        |  cosmic-screenshot|
                        |  grim / slurp     |
                        |  legacy tools     |
                        |  portal           |
                        +-------------------+
                                  |
                        +---------v---------+
                        | Wayland Compositor |
                        +-------------------+

Troubleshooting

Start here, whatever the symptom. Call the describe_environment tool: it reports which binaries were found, which portal interfaces are on the bus, which backend was selected for capture, keyboard and pointer, and, when one is unavailable, exactly what each candidate would have needed.

Everything reports as unavailable when launched from an MCP client

• Almost always the stripped environment. The server recovers XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and WAYLAND_DISPLAY on its own, but it refuses to guess between several compositor sockets. Set WAYLAND_DISPLAY in the env block of your client config.
• Check the log (/tmp/wayland-mcp.log) for the line listing what it recovered.

Input control not working

• No sudo step exists any more. If you are looking for setup.sh, read the warning under Input control setup.
• If a permission dialog never appears and nothing happens, your desktop may have no RemoteDesktop portal (wlroots compositors do not ship one). Install wtype for keyboard, ydotool for pointer.
• If the permission was denied once, the stored token is stale: delete $XDG_STATE_HOME/wayland-mcp/remote-desktop-token and try again.
• Verify against a real window rather than guessing: python scripts/verify_input.py clicks, types and scrolls, then reads back what the widgets actually received. Do not touch the machine while it runs.

Screenshots failing

• On COSMIC, a grim older than 1.5 cannot work: the compositor implements only ext-image-copy-capture. Selection already accounts for this; if you forced WAYLAND_MCP_CAPTURE_BACKEND=grim, unset it.
• Install a portal (xdg-desktop-portal plus the backend for your desktop) and PyGObject for the universal fallback.

Typing produces the wrong characters

• You are on a non-US layout with a keycode-based backend (ydotool, evemu), which types positions rather than characters. The portal backend types by keysym and is layout independent.

VLM analysis not working

• Vision is optional: capture and input do not need it. The analysis tools say precisely which variable to set when no provider is configured.
• Confirm the key matches the provider you selected with VLM_PROVIDER.

Server won't start

• Check Python: python3 --version (needs 3.10+).
• Reinstall: uv pip install -e ".[dev]", then pytest. The test suite runs headless and does not need a graphical session.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for guidelines.

Project structure

wayland-mcp/
├── wayland_mcp/            # Main package
│   ├── server_mcp.py       # MCP server implementation
│   ├── screen_utils.py     # Screenshot and VLM analysis
│   ├── mouse_utils.py      # Pointer control functions
│   ├── keyboard_utils.py   # Keyboard input handling
│   ├── chain_processor.py  # Action chain parser
│   ├── session_env.py      # Recovers session vars MCP clients strip
│   └── backends/           # Capability-selected capture and input backends
│       ├── base.py         # Capabilities probe and backend interfaces
│       ├── detect.py       # Selection by priority and requirements
│       ├── portal.py       # Shared GDBus plumbing for XDG portals
│       ├── capture_*.py    # cosmic-screenshot, grim, legacy tools, portal
│       └── input_*.py      # portal RemoteDesktop, wtype, ydotool, evemu
├── tests/                  # Headless tests: selection, keycodes, session env
├── scripts/
│   ├── verify_input.py     # End-to-end input check against a real window
│   └── legacy-evemu-setup.sh  # Documents how to undo the upstream setup.sh
├── README.md               # This file
├── CONTRIBUTING.md         # Contribution guidelines
└── pyproject.toml          # Package metadata

License

GPL-3.0. See LICENSE for the full text.

Copyright (C) 2026 wayland-mcp contributors.