agent-aid
A small Python library and CLI to control the local Windows desktop from AI
agents. Open Interpreter, GPT, Claude, or your own agent — call agent-aid to
read the screen, drive mouse and keyboard, manage windows, and find UI
elements by their accessibility name (no coordinate guessing). Two
dependencies: mss (screen capture) and uiautomation (UI tree).
pip install agent-aid
With uv (recommended — installs Python automatically if needed):
uv tool install agent-aid --python 3.11
Quick look
agent-aid health
agent-aid state
agent-aid screenshot active_window=true save_path=captures/active.png include_base64=false
agent-aid click x=500 y=300
agent-aid type text="hello"
agent-aid press keys=ctrl+s
agent-aid focus_window title_fragment=Chrome
agent-aid open target=https://example.com
List every route:
agent-aid --list
Print the full AI-oriented usage spec to stdout (markdown):
agent-aid --readme
Capabilities
- Screenshots: full desktop, single monitor, active window, specific
hwnd, or rectangular region - PNG
sha256hash returned for every screenshot (use withwait_screen_change) - Mouse: click, double-click, right-click, drag, move, scroll (vertical/horizontal), hold buttons
- Drag with optional
pre_hold_ms,post_hold_ms,stepsfor finicky shape/preview UIs - Keyboard: short text, hotkeys (
ctrl+shift+a), modifier hold/release - Clipboard:
clipboard text=...+press keys=ctrl+vfor long pastes - Windows: find, focus, minimize/maximize/restore/close, move/resize, hide/show
- UI Automation: find/click/read/write elements by accessibility name (no coordinate guessing)
- System: list processes, open file/URL/
shell:target, read pixel color, query state - Verify:
wait_screen_change,wait_pixel,wait_window batchfor atomic multi-step flows in a single CLI call
UI Automation (recommended over raw coordinates)
# Find an element by partial name match (case-insensitive)
agent-aid ui_find name="Save"
agent-aid ui_find name="Yapıştır" control_type=Button hwnd=12345
# Find every match (use index=N to pick the n-th, e.g. for repeated names)
agent-aid ui_find_all control_type=Button hwnd=12345 limit=20
agent-aid ui_click name="Delete" index=1 # 2nd "Delete"
# Find + click in one call
agent-aid ui_click name="OK"
# Read the text/value of a control (uses ValuePattern → TextPattern → Name)
agent-aid ui_text automation_id="UrlBar" hwnd=12345
# Append text into an edit/combo via UIA (safe-by-default: does NOT clear
# existing content, inserts at cursor). Full Unicode, no key races.
agent-aid ui_set_value automation_id="UrlBar" hwnd=12345 text="https://example.com"
# Replace the entire field — opt-in destructive mode
agent-aid ui_set_value automation_id="UrlBar" hwnd=12345 text="..." clear=true
# Dump the accessibility tree (filter by control_type to keep small)
agent-aid ui_dump hwnd=12345 max_depth=4 limit=100
agent-aid ui_dump hwnd=12345 max_depth=10 control_type=ListItem
Works on Paint, Office, Edge, Chrome, native Win32, and most modern UWP apps.
Selector args: name (partial), control_type, automation_id (exact),
class_name (partial), hwnd (root scope), index, max_depth, timeout_ms.
Targeting windows / coordinates
All coordinates are physical screen pixels. To work relative to a specific window:
agent-aid click x=120 y=80 relative_to=active_window
agent-aid click x=120 y=80 hwnd=123456
Use as a Python library
from agent_aid import core
core.set_dpi_aware()
print(core.active_window())
core.click(800, 500)
core.type_text("hello")
Same capabilities as the CLI — just call the core module directly from your
Python code.
Argument formats
# key=value (shortest)
agent-aid click x=500 y=300 button=left
# JSON (for nested fields)
agent-aid screenshot '{"region":{"left":0,"top":0,"width":800,"height":600},"save_path":"r.png"}'
# Pretty-print output
agent-aid --pretty state
Practical AI agent flow
agent-aid state— see what you're looking atagent-aid screenshot active_window=true save_path=captures/now.png include_base64=false- Inspect the image, choose target coordinates
- Act:
click/type/press/clipboard - Verify: another
screenshotorwait_screen_change
Safety notes
- Sends real mouse and keyboard input — types into the focused window.
clipboardoverwrites the user's clipboard.openlaunches a Windows target (same effect as a user double-click).window/manage closepostsWM_CLOSE— apps with unsaved data may prompt.- After any action, prefer to verify with
wait_*or a freshscreenshot.
License
MIT.
Metadata
Release files for agent_aid 1.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_aid-1.7.0.tar.gz | 26.4 kB | Without provenance | Details for agent_aid-1.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_aid-1.7.0-py3-none-any.whl | Python 3 | none | any | Details for agent_aid-1.7.0-py3-none-any.whl |
Total release size: 54.0 kB
Release files / agent_aid-1.7.0.tar.gz
| Download URL | agent_aid-1.7.0.tar.gz |
|---|---|
| Size | 26.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
78dfe94a0bb3daf5e840fc2fdc49f6eace25a0a2f6d0d541bf6f6d721a380354
|
|
BLAKE2b-256 checksum How to use checksums |
44c2b97c0f022cb05bed98cc1df472baee384c47a3de00df5dc92b3ccf1ae3d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / agent_aid-1.7.0-py3-none-any.whl
| Download URL | agent_aid-1.7.0-py3-none-any.whl |
|---|---|
| Size | 27.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
99fd896fa3fa365aa672e4533713167d14aa5fa2ca5e9955c3858f1cb58d52a3
|
|
BLAKE2b-256 checksum How to use checksums |
e29e7c61869c07c148929f7604ac57728cc5aef93984a81aed77e92f73b9afb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for agent_aid 1.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.