wayland-computer-use

Desktop control for AI agents on native Wayland, where xdotool and pyautogui do not work and cannot be made to work.

One file, no dependencies beyond the portal client. Extracted from a working Telegram agent that drives a KDE Plasma 6 desktop.

wayland-computer screenshot
wayland-computer click 640 480
wayland-computer type "привет, мир"      # this is the part that usually fails
wayland-computer key ctrl+alt+t
wayland-computer focus chromium

---

Why this exists

Wayland is deliberately built so that a client cannot read another window's pixels or deliver synthetic input to it. That is a security feature, and it means the entire X11 automation stack is dead on a modern session:

• Ubuntu 26.04 LTS (23 April 2026) is the first Ubuntu LTS whose default desktop ships without a GNOME-on-Xorg session to fall back to.
• RHEL 10 removed the X.Org server, keeping only XWayland — it is listed under Removed features in the release documentation.
• GNOME 49 disabled the GNOME-on-X11 session (GDM still launches other X11 sessions); the X11 backend itself was removed in GNOME 50.

So anything built on xdotool, pyautogui or XTEST — which is most computer-use agents, most desktop RPA, and most desktop QA harnesses — silently stops working. This library takes the paths that still work.

What is actually solved here

Unicode input, including Cyrillic

The one that costs everyone else a day. On Wayland the working path is to send keysyms, not scancodes, and there is a universal rule for any codepoint:

def char_keysym(ch: str) -> int:
    cp = ord(ch)
    if 0x20 <= cp <= 0x7E:      # basic Latin maps to itself
        return cp
    return 0x01000000 | cp      # the portals' universal Unicode rule

Four lines, and Cyrillic, Greek, CJK and everything else types correctly.

For comparison, as of August 2026: ydotool type handles US-ASCII only against a hardcoded layout (the UTF-8 patch has been sitting unmerged since 7 July); kwin-mcp looks each character up in a table built from four rows of US QWERTY and, on a miss, skips the character and reports success; Handy needs a separate kwtype binary on KDE because wtype has no zwp_virtual_keyboard_manager_v1 there.

Input through the portal, with the consent dialog shown once

xdg-desktop-portal RemoteDesktop for keyboard and pointer, ScreenCast for absolute coordinates. The permission dialog appears on first use; after that a restore_token is persisted and reused. No root, no uinput group, no udev rules, no daemon.

Window focus that works when nothing else does

The agent opens a browser, starts typing the query — and the focus is still on Telegram, so the query goes into the chat. On Wayland you cannot fix this with wmctrl or xdotool: the compositor does not hand other windows to anyone.

The only door is KWin scripting over D-Bus (loadScript / run / unloadScript). One catch worth knowing before you try it yourself: print() inside a KWin script goes to journald, not to qdbus stdout, and there is no other channel out of the script. This library reads the answer back from the journal.

Screenshots that fail loudly instead of hanging

Spectacle with a hard timeout and retries — a hung compositor would otherwise block the call forever. Screenshots land in a private 0700 directory with unique names, and the last ~20 are kept.

Output designed for a model, not a human

Every command prints one line. Errors print one line on stderr, never a traceback. This is not tidiness: a 1400-character Python stack full of absolute paths costs an agent an entire turn, and it learns nothing from it. Exit codes are part of the contract — 0 done, 2 bad arguments, 3 not found, 1 everything else — so a caller can branch without parsing prose.

Human-paced pointer and typing

The cursor travels along an eased arc with slight jitter; typing has a live rhythm with longer pauses after spaces and punctuation. Two reasons this is not decoration: a teleporting cursor is unwatchable in a demo, and some toolkits only fire hover state on real motion. WCU_INPUT_STYLE=instant turns it off.

---

Install

Not on PyPI yet — install from the repository:

pip install "wayland-computer-use[linux] @ git+https://github.com/wh0ami3/wayland-computer-use"

Requirements on Linux: a Wayland session, xdg-desktop-portal with a backend that implements RemoteDesktop and ScreenCast (KDE and GNOME both do), and spectacle for screenshots on KDE.

Windows is supported through pyautogui (pip install wayland-computer-use[windows]).

Use from Python

import asyncio
from wayland_computer_use import Computer

async def main():
    c = Computer()
    path = await c.screenshot()
    await c.click(640, 480)
    await c.type_text("привет")
    await c.key("ctrl+s")
    ok, detail = await c.focus("chromium")

asyncio.run(main())

Use from the shell

Built to be called by an agent as a tool.

command	what it does
screenshot [path.png]	capture the screen, print the path
click X Y [left|right|middle|double]	click at absolute coordinates
move X Y	move the cursor
type "text"	type text, Unicode included
key "ctrl+alt+t"	press a combination
scroll DY	scroll, negative scrolls up
focus "substring"	focus a window; on miss, prints the open windows
find "description"	locate an element (needs an aimer, see below)
clickon "description"	locate and click

Aimer plugin

find and clickon need a vision model to turn "the blue Save button" into coordinates. Rather than bundling one, point WCU_AIMER at your own:

export WCU_AIMER="my_module:aim"     # aim(image_path, target) -> (x, y) | None

Anything works — a local Qwen-VL over an Ollama endpoint, a hosted model, or template matching. Not set, or no hit, and the command exits 3, so the caller can fall back to aiming by itself.

Configuration

variable	default	meaning
WCU_HOME	~/.wayland-computer-use	state: restore token, screenshots, last pointer position
WCU_INPUT_STYLE	human	instant disables the glide and typing rhythm
WCU_AIMER	unset	module:function for find / clickon

Status and limits

Beta. Used daily on Arch Linux with KDE Plasma 6 on Wayland; that is the only configuration tested end to end.

Known limits, stated plainly:

• Screenshots go through Spectacle, so on KDE only. CaptureWindow on org.kde.KWin.ScreenShot2 returns NoAuthorized without a X-KDE-DBUS-Restricted-Interfaces entry in a .desktop file (KDE bug 446628), so single-window capture is not available — capture full screen and crop by the window's frameGeometry.
• Window focus is KWin-specific. GNOME and wlroots compositors need their own backend; the interface is there, the implementations are not.
• No accessibility tree yet. AT-SPI2 would let an agent read window state as text instead of pixels, which is cheaper in tokens and far more accurate than aiming at a screenshot. That is the next thing worth building.
• The portal consent dialog cannot be avoided on first run by design. If you need it gone entirely, KWin exposes a private EIS interface that skips it.

How fast does it type?

A portal-based input path in another project was measured at 3-4 characters per second and its author called it unusable, so this is worth measuring rather than claiming. bench/typing_speed.py reports the transport ceiling and the real type_text rate, for ASCII and non-ASCII, on your machine:

python bench/typing_speed.py

Two things keep the rate up here: paced typing is a deliberate default that WCU_INPUT_STYLE=instant removes, and any text over 120 characters skips per-character typing entirely and goes through the clipboard in one paste.

Contributions welcome, especially a GNOME backend and an AT-SPI2 reader.

License

MIT.