pctr

Element-based Windows UI Automation CLI for AI agents.

Instead of screenshotting and guessing pixel coordinates, pctr finds controls by their name / control type / automation id and acts on them. Window moves, DPI changes, and layout shifts don't break your script.

Built on pywinauto (UI Automation) with pyautogui / pydirectinput for raw mouse and keyboard input. On top of UIA it also offers OCR (Windows.Media.Ocr) and YOLO-World open-vocabulary detection for text and objects that aren't exposed as controls.

Install

pip install pctr
pip install "pctr[vision]"   # adds ultralytics for `pctr look`

Windows only.

Commands

Common filters: --title (regex), --name (regex), --control-type (Button, Edit, Pane, MenuItem, …), --auto-id, --nth, --process.

click, set, wait accept any selector - --name is not required. tree/find require a window selector (--title or --process).

The loop

pctr tree  --title "Notepad" --limit 60      # see what's clickable
pctr click --title "Notepad" --name "^File$"
pctr set   --title "Notepad" --name "Text Editor" --text "hello"

Virtual desktops

pctr desktop list                       # desktops (#index, guid, [current], window count)
pctr desktop windows --desktop 1        # windows on desktop #1
pctr desktop where --title "App"        # which desktop a window is on
pctr desktop on-current --title "App"   # True / False
pctr desktop next | prev | new | close  # switch (moves the active desktop)
pctr desktop move-window --title RE --to N

• Switching uses the standard Win+Ctrl hotkeys and moves the active desktop (you included). Add --direct if a game has focus.
• Cross-desktop control: UIA actions (click --method invoke, set) reach windows on another desktop without switching (accessibility layer, not screen input). Raw mouse/keys only affect the active desktop.
• move-window is blocked on Windows 11 (third-party MoveWindowToDesktop returns Access denied); it works on Windows 10.

Exit codes

• 0 success, 1 no match / window not found, 2 usage/validation error.
• Errors print one error: … line - no tracebacks.

Notes

• Prefer set (ValuePattern) over type for text fields - it can't drop or reorder characters. type's element path uses pywinauto type_keys, which interprets { } + ^ % ~ ( ) as key syntax, so it is not literal.
• Global type / keys go to the OS-focused window - pctr focus first.
• Prefer pctr hotkey --keys ctrl+s over keys "^s"; the ^ modifier is fragile through shells.
• ocrfind/ocrclick match one word at a time (Windows OCR tokenizes words).
• look reads its model from --model or the PCTR_YOLO_MODEL env var; progress bars go to stderr so stdout stays parseable.
• Qt apps expose a rich tree including embedded webviews. Electron apps expose only the window frame (use focus + global keys, or drive them over CDP).
• Games ignore synthetic input - pass --direct to mouse/key commands and focus the game first.

License

MIT