pctr

Element-based Windows UI Automation CLI for AI agents.

Instead of screenshotting and guessing pixel coordinates, pctr finds controls by their name / control type / automation id and acts on them. Window moves, DPI changes, and layout shifts don't break your script.

Built on pywinauto (UI Automation) with pyautogui / pydirectinput for raw mouse and keyboard input. On top of UIA it also offers OCR (Windows.Media.Ocr) and YOLO-World open-vocabulary detection for text and objects that aren't exposed as controls.

Driven entirely by pctr itself - list windows, map virtual desktops, dump a control tree, set a field, read it back with OCR, screenshot it, and YOLO-World detect. Recorded with OBS, which pctr also piloted.

Install

pip install pctr             # core (UIA + OCR)
pip install "pctr[vision]"   # + ultralytics, for `pctr look`
pip install "pctr[mcp]"      # + MCP SDK, for the MCP server
pip install "pctr[all]"      # everything

Windows only.

MCP server

pctr ships a Model Context Protocol server so MCP clients (Claude Desktop, Cursor, opencode, ...) can drive the Windows desktop directly:

pip install "pctr[mcp]"

Add it to your client config (stdio):

{ "mcpServers": { "pctr": { "command": "pctr-mcp" } } }

(or "command": "python", "args": ["-m", "pctr.mcp_server"])

Tools exposed: pctr_windows, pctr_tree, pctr_find, pctr_click, pctr_set, pctr_type, pctr_keys, pctr_hotkey, pctr_focus, pctr_wait, pctr_shot, pctr_ocr, pctr_ocrfind, pctr_ocrclick, pctr_look, pctr_desktop.

Or run/manage it from the CLI:

pctr mcp status                                          # SDK + server state
pctr mcp start --transport streamable-http --port 8765   # background HTTP server
pctr mcp restart | pctr mcp stop
pctr mcp help

stdio runs in the foreground (what a client spawns); sse / streamable-http run in the background, reachable at http://127.0.0.1:8765/mcp (or …/sse).

Commands

Global flags: --json (machine-readable output) and --file FILE (run a batch of commands from a file).

Common filters: --title (regex), --name (regex), --control-type (Button, Edit, Pane, MenuItem, …), --auto-id, --nth, --process.

click, set, wait accept any selector - --name is not required. tree/find require a window selector (--title or --process).

The loop

pctr tree  --title "Notepad" --limit 60      # see what's clickable
pctr click --title "Notepad" --name "^File$"
pctr set   --title "Notepad" --name "Text Editor" --text "hello"

JSON output

Add --json (global, or per-command) for machine-readable output from windows, tree, find, ocr, ocrfind, look, and desktop:

pctr windows --json
# [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]

pctr desktop list --json
# [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]

pctr look --for "a car" --json
# [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]

Progress and warnings go to stderr, so stdout stays valid JSON.

Batch files

pctr --file script.txt          # one command per line; # comments and blanks skipped
pctr --json --file script.txt   # --json applies to every line

Stops on the first failing line and reports its line number.

Virtual desktops

pctr desktop list                       # desktops (#index, guid, [current], window count)
pctr desktop windows --desktop 1        # windows on desktop #1
pctr desktop where --title "App"        # which desktop a window is on
pctr desktop on-current --title "App"   # True / False
pctr desktop next | prev | new | close  # switch (moves the active desktop)
pctr desktop move-window --title RE --to N

• Switching uses the standard Win+Ctrl hotkeys and moves the active desktop (you included). Add --direct if a game has focus.
• Cross-desktop control: UIA actions (click --method invoke, set) reach windows on another desktop without switching (accessibility layer, not screen input). Raw mouse/keys only affect the active desktop.
• move-window is blocked on Windows 11 (third-party MoveWindowToDesktop returns Access denied); it works on Windows 10.

Exit codes

• 0 success, 1 no match / window not found, 2 usage/validation error.
• Errors print one error: … line - no tracebacks.

Notes

• Prefer set (ValuePattern) over type for text fields - it can't drop or reorder characters. type's element path uses pywinauto type_keys, which interprets { } + ^ % ~ ( ) as key syntax, so it is not literal.
• Global type / keys go to the OS-focused window - pctr focus first.
• Prefer pctr hotkey --keys ctrl+s over keys "^s"; the ^ modifier is fragile through shells.
• ocrfind/ocrclick match one word at a time (Windows OCR tokenizes words).
• look reads its model from --model or the PCTR_YOLO_MODEL env var; progress bars go to stderr so stdout stays parseable.
• Qt apps expose a rich tree including embedded webviews. Electron apps expose only the window frame (use focus + global keys, or drive them over CDP).
• Games ignore synthetic input - pass --direct to mouse/key commands and focus the game first.

License

MIT