ElfDeckAI

Virtual keyboard, gamepad and mouse for AI agents on Linux and the Steam Deck.

ElfDeckAI lets an AI agent operate a real Linux machine through its input devices. Three small daemons create virtual devices on /dev/uinput, and an MCP server exposes them, together with a screenshot tool, as tools any MCP client can call:

• ElfKey: a virtual keyboard
• ElfPad: a virtual Xbox 360-class gamepad (buttons, analog triggers, two sticks, D-pad)
• ElfMouse: a virtual mouse (relative pointer, buttons, wheel)

Applications, games, Steam Input and the compositor see these as ordinary hardware. Unlike emulator-scripting approaches, nothing depends on a particular emulator or game: ElfDeckAI has been used to play Pokémon FireRed in RetroArch on a physical Steam Deck from the intro through the first rival battle, with the agent seeing only screenshots and acting only through ElfKey. See notes/driving.md for what was learned.

The daemons and server are single-file Python programs that use only the standard library.

Quick start: Steam Deck

On the Deck, switch to Desktop Mode, open Konsole and run:

curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/install-on-deck.sh | sh

The installer works entirely inside your home directory (no sudo):

1. installs the elfdeckai package for your user,
2. installs and starts elfkey, elfpad and elfmouse as systemd user services,
3. checks that all three virtual devices are present, and
4. prints the MCP configuration for your PC and the commands to enable SSH.

It is safe to re-run; it upgrades in place.

For a guided setup, give AI_SETUP_GUIDE.md to your AI assistant. It is written for an assistant to follow and walks through the Deck side, the PC side and troubleshooting.

Quick start: any Linux machine

pip install elfdeckai

elfkey &
elfpad &
elfmouse &

echo "key x 150"     > /tmp/elfkey.fifo     # tap X for 150 ms
echo "press a"       > /tmp/elfpad.fifo     # tap A
echo "move 300 -100" > /tmp/elfmouse.fifo   # move the pointer
echo "click left"    > /tmp/elfmouse.fifo   # left click

grep -E "ElfKey|ElfPad|ElfMouse" /proc/bus/input/devices   # confirm the devices exist

Without pip, clone the repository and run python3 elfkey.py (and so on) directly. Runtime state lives in /tmp and is cleared on reboot.

/dev/uinput is user-writable on the Steam Deck by default. On other distributions, install the udev rule first (see Installation).

MCP server

elfdeckai-mcp is a Model Context Protocol server (stdio transport) that exposes 21 tools:

screenshot captures the screen of the machine the server runs on and returns a PNG, so an agent can work in a look, act, look again loop without a separate video stream. It supports the Steam Deck in both Gaming Mode (through gamescope) and Desktop Mode (KDE Plasma), wlroots compositors and X11. Screens larger than 1280 pixels on their long side are scaled down before sending (set ELFDECKAI_SCREENSHOT_MAX to change the limit, or 0 to turn it off), and the result includes the image's size.

mouse_click_at and mouse_move_to take coordinates in the pixels of the most recent screenshot, so an agent can click exactly what it sees, whatever the screen's real resolution or the scaling applied. They assume a single display.

Running on the same machine as the daemons:

{"mcpServers": {"elfdeckai": {"command": "elfdeckai-mcp"}}}

Running on a Steam Deck from an AI client on your PC, over SSH:

{"mcpServers": {"elfdeckai": {
    "command": "ssh",
    "args": ["deck@DECK-IP-ADDRESS", "/home/deck/.local/bin/elfdeckai-mcp"]}}}

This needs key-based SSH login from the PC to the Deck, because the MCP client cannot answer a password prompt; AI_SETUP_GUIDE.md covers the setup. Without pip, use "command": "python3", "args": ["/path/to/elfdeckai_mcp.py"].

Every tool call is validated against the tool's schema and the daemon's own parser before anything is written to a FIFO. If a daemon is not running, the call returns an error immediately instead of blocking.

Command reference

ElfKey (/tmp/elfkey.fifo)

Keys: a–z, 0–9, arrows, enter, esc, space, tab, backspace, shift, rshift, ctrl, rctrl, alt, ralt, capslock, f1–f10, home, end, pageup, pagedown, insert, delete and common punctuation. The full table is KEYS in elfkey.py.

ElfPad (/tmp/elfpad.fifo)

Buttons: a b x y lb rb l2 r2 select start guide l3 r3, plus D-pad directions up down left right.

ElfPad identifies as an Xbox 360 controller, so SDL, Steam Input and RetroArch map it without configuration. l2 and r2 drive the analog trigger axes, as on the real controller; there are intentionally no digital trigger buttons, because declaring them shifts Select, Start and Guide off the standard button indices.

ElfMouse (/tmp/elfmouse.fifo)

move is relative: negative dy moves up. Its units are motion counts, not guaranteed screen pixels, because pointer acceleration applies. Each command is limited to ±10,000 counts and ±120 wheel ticks.

moveto uses a second virtual device, ElfMouse Absolute, which compositors treat like a virtual machine's tablet pointer, so the position is exact. Buttons are pressed on whichever device moved last.

Safety model

Because the caller may be a language model, the daemons are built so that bad input cannot leave the machine in a bad state:

• Malformed input is refused, not fatal. Every line passes a strict parser; invalid commands are logged and ignored.
• Taps are bounded. A timed hold lasts at most 5 seconds. Longer holds use down/up.
• Nothing stays stuck. Timed actions restore in a finally block, held inputs are tracked, and on quit, SIGTERM or SIGINT every input is released before the device is removed. key_release, mouse_release and pad_reset release everything on demand.
• Owner-only access. Each FIFO is created with mode 0600.
• Audit log. Every accepted command is written to stderr (the journal, when run as a service).
• Robust protocol handling. The MCP server returns proper JSON-RPC errors for malformed or oversized requests and never blocks on a stopped daemon.

Scope: these are real OS-level input devices. They act on whichever window has focus, including terminals and browsers. ElfDeckAI keeps input bounded, validated and releasable; deciding where an agent is allowed to act is up to the operator. Only run agents you trust, in sessions you can supervise.

Comparison with other approaches

Installation

Requirements: 64-bit Linux with /dev/uinput and Python 3.9 or newer.

Where /dev/uinput is not writable by your user, install the udev rule once:

sudo cp 70-elfdeckai-uinput.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
getfacl /dev/uinput   # your user should appear in the ACL

To run the daemons as systemd user services after pip install --user elfdeckai (the unit files expect the commands in ~/.local/bin):

mkdir -p ~/.config/systemd/user
cp systemd/*.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now elfkey.service elfpad.service elfmouse.service

Development

pip install -e ".[test]"
python3 -m pytest

The test suite covers each daemon's input parser, device behaviour against a fake uinput device, and the MCP server over real JSON-RPC. No /dev/uinput is required to run it.

Roadmap

See ROADMAP.md.

Changelog

See CHANGELOG.md.

License

MIT. See LICENSE.

Author

Created and maintained by Garth Elfstrom (@Garthnesss). Issues and pull requests are welcome.