AutoControl
AutoControl is a cross-platform GUI automation framework for Python. It drives the mouse and keyboard, finds things on screen (template matching, OCR, the OS accessibility tree, or a vision model), records and replays flows, and runs them from JSON action files — on Windows, macOS, Linux (X11 and Wayland), Android, and iOS.
Every capability ships three ways: a Python API, an AC_* action command usable
from JSON files / CLI / servers, and a GUI tab. Nothing is GUI-only.
Why AutoControl
- One API, six platforms.
wrapper/platform_wrapper.pypicks the backend at import time; your script does not change between Windows, macOS, X11, and Wayland. - Scriptable without Python. 767
AC_*commands cover the whole feature set, so a JSON file can do anything the library can — including loops, branches, try/catch, macros, and variables. - Headless by default.
import je_auto_controlnever loads Qt. The GUI is an optional extra that wraps the same headless core. - Locate things four ways. Template matching, OCR, the accessibility tree, and a vision-language model — composable through anchor locators and self-healing fallbacks.
- Light dependency floor. The REST server, JSON Schema validator, JWT, TOTP, WebSocket framing, ACME client, USB/IP protocol, and Prometheus metrics are all standard-library implementations. Heavy things are opt-in extras.
Installation
pip install je_auto_control # core
pip install je_auto_control[gui] # + PySide6 desktop app
Optional extras, installed only when you need them:
| Extra | Enables |
|---|---|
gui |
PySide6 desktop application (48 tabs) |
webrtc |
WebRTC remote desktop, USB passthrough (aiortc, av) |
signaling |
Standalone signaling / rendezvous server (fastapi, uvicorn) |
discovery |
mDNS / Zeroconf LAN host discovery |
pdf / office |
PDF and Excel / Word / PowerPoint reading |
fuzzy / locale |
rapidfuzz matching, babel locale parsing |
s3 / audio |
S3 artifact store, system volume control |
Requirements: Python ≥ 3.10. On Linux, install build prerequisites first:
sudo apt-get install cmake libssl-dev
OCR, VLM, and LLM backends (pytesseract, easyocr, paddleocr, anthropic,
openai) are loaded on demand — install whichever you actually use.
60-second quick start
1. As a Python library
import je_auto_control as ac
ac.set_mouse_position(500, 300)
ac.click_mouse("mouse_left")
ac.write("Hello World")
ac.hotkey(["ctrl_l", "s"])
x, y = ac.locate_image_center("save_button.png", detect_threshold=0.9)
ac.click_text("Submit") # OCR
ac.click_accessibility_element(name="OK") # accessibility tree
ac.click_by_description("the green Submit button") # vision model
ac.screenshot("shot.png", screen_region=[0, 0, 800, 600])
2. As a JSON action file — flow.json
[
["AC_set_var", {"name": "user", "value": "alice"}],
["AC_locate_and_click", {"image": "login.png", "mouse_keycode": "mouse_left"}],
["AC_write", {"write_string": "${user}"}],
["AC_retry", {"max_attempts": 3, "body": [
["AC_wait_text", {"target": "Welcome", "timeout": 10}]
]}],
["AC_assert_text", {"text": "Welcome"}],
["AC_generate_html_report", {"html_name": "report"}]
]
je_auto_control run flow.json --var user=bob
je_auto_control run flow.json --dry-run # list the steps without touching the mouse
3. As a desktop app
pip install je_auto_control[gui]
python -m je_auto_control # or: je_auto_control.start_autocontrol_gui()
Record a flow, edit it in the visual Script Builder, and save it as the same JSON format the CLI runs.
Capability overview
Every row works headlessly. "GUI tab" is where the same feature surfaces in the desktop app; tab commands live in the window's Actions menu.
| Capability | Python API | AC_* command |
GUI tab |
|---|---|---|---|
| Mouse | click_mouse, set_mouse_position, mouse_scroll |
AC_click_mouse |
Auto Click |
| Keyboard | write, hotkey, type_keyboard |
AC_write, AC_hotkey |
Auto Click |
| Screen & pixels | screenshot, screen_size, get_pixel |
AC_screenshot |
Screenshot |
| Image matching | locate_image_center, locate_and_click |
AC_locate_and_click |
Image Detect |
| OCR text | click_text, wait_for_text, read_text_in_region |
AC_click_text, AC_wait_text |
OCR Reader |
| Accessibility tree | find_accessibility_element, click_accessibility_element |
AC_a11y_find, AC_a11y_click |
Accessibility |
| Vision-model locator | locate_by_description, click_by_description |
AC_vlm_locate, AC_vlm_click |
VLM |
| Anchor locator | — | AC_anchor_click, AC_anchor_locate |
— |
| Self-healing locators | self_heal_click, self_heal_locate |
AC_self_heal_click |
Self-Healing |
| Natural-language planner | plan_actions, run_from_description |
AC_llm_plan |
LLM Planner |
| Computer-use agent | AgentLoop, run_agent |
AC_run_agent |
Computer Use |
| Record & replay | record, stop_record |
AC_record, AC_stop_record |
Record |
| JSON scripting | execute_action, execute_files |
all 767 commands | Script, Script Builder |
| Variables & flow control | execute_action_with_vars |
AC_set_var, AC_loop, AC_for_each, AC_try, AC_retry |
Variables |
| Data-driven runs | — | AC_for_each_row (CSV / JSON / SQLite / Excel) |
Data Sources |
| Assertions | assert_text, assert_image |
AC_assert_text + 20 more |
Assertions |
| Test suites | run_suite |
AC_run_suite |
Test Suites |
| Scheduler (interval + cron) | default_scheduler |
— | Scheduler |
| Global hotkeys | default_hotkey_daemon |
— | Hotkeys |
| Event triggers | default_trigger_engine |
AC_email_trigger_add |
Triggers, Webhooks, Email |
| Window management (Windows) | list_windows, focus_window |
AC_focus_window, AC_snap_window |
Window Manager |
| Clipboard (text + image) | get_clipboard, set_clipboard, get_clipboard_image, set_clipboard_image |
AC_clipboard_get, AC_clipboard_set, AC_clipboard_get_image, AC_clipboard_set_image |
— |
| Remote desktop | RemoteDesktopHost, RemoteDesktopViewer |
AC_start_remote_host, AC_remote_connect |
Remote Desktop |
| USB enumeration & passthrough | list_usb_devices, enable_usb_passthrough |
AC_usb_* (16 commands) |
USB Devices, USB Share |
| Secrets vault | default_secret_manager |
AC_secret_set + ${secrets.NAME} |
Secrets |
| Reports (HTML / JSON / XML) | generate_html_report |
AC_generate_html_report |
Report |
| Run history | — | — | Run History |
| Metrics & tracing | default_metric_registry, render_metrics_text |
— | — |
| Diagnostics | run_diagnostics |
AC_diagnose |
Diagnostics |
| Test-code generation | generate_code |
— | — |
Beyond this table, utils/ holds 308 headless packages covering assertions, resilience,
data quality, i18n auditing, redaction, governance, observability, and more. The full
per-module map is in architecture_explore.md.
Command-line interface
je_auto_control run script.json [--var name=value] [--dry-run]
je_auto_control validate script.json # alias: lint
je_auto_control fmt script.json [--check]
je_auto_control list-commands [--filter mouse] [--json]
je_auto_control record out.json [--duration 5]
je_auto_control codegen script.json --target pytest -o test_flow.py
je_auto_control failure-bundle failure.zip --error "login timed out"
je_auto_control list-jobs
je_auto_control start-server --port 9938 # TCP socket server
je_auto_control start-rest --port 9939 # REST API
je_auto_control version
--var name=value is parsed as JSON when possible (count=10 becomes an int),
otherwise kept as a string. The legacy python -m je_auto_control -e file.json
entry point still works.
Servers and integrations
| Surface | Start it with | Notes |
|---|---|---|
| MCP server | je_auto_control_mcp (stdio) or AC_start_mcp_http_server |
670 tools for Claude Desktop / Claude Code / custom tool loops. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. |
| REST API | je_auto_control start-rest |
Bearer token, per-IP rate limit + lockout, SQLite audit hook, /metrics, /openapi.json, /docs Swagger UI, /dashboard. |
| TCP socket server | je_auto_control start-server |
Newline-framed JSON action lists. Binds 127.0.0.1 by default. |
| pytest plugin | installed automatically | Fixtures plus a Gherkin step library for pytest-bdd / behave. |
| Language server | python -m autocontrol_lsp.server |
Completion and diagnostics for AC_* action JSON, generated from the live command table. |
| Remote desktop | RemoteDesktopHost / GUI |
TCP, WebSocket, or WebRTC; TOTP, trust list, TURN config, file/clipboard/audio sync. |
All servers bind to 127.0.0.1 unless you opt in explicitly.
How the remote-desktop wire protocol works
Worth knowing before you expose a host, and described nowhere else in the docs. The default transport is length-prefixed framing over raw TCP — no extra dependencies — and it opens with an HMAC-SHA256 challenge/response handshake: a viewer that fails auth is dropped before it is sent a single frame. JPEG frames are encoded at the configured FPS and quality and handed to authenticated viewers through a shared latest-frame slot, so a slow viewer drops frames instead of stalling the rest. Viewer input arrives as JSON and is validated against an allow-list of actions before being applied through the ordinary input wrappers, so a viewer cannot invent new operations.
# Be remoted — start a host and hand the token + port to whoever views you
from je_auto_control import RemoteDesktopHost
host = RemoteDesktopHost(token="hunter2", bind="127.0.0.1",
port=0, fps=10, quality=70)
host.start()
print("listening on", host.port, "viewers:", host.connected_clients)
# Control another machine — connect a viewer and send input
from je_auto_control import RemoteDesktopViewer
viewer = RemoteDesktopViewer(host="10.0.0.5", port=51234, token="hunter2",
on_frame=lambda jpeg: ...)
viewer.connect()
viewer.send_input({"action": "mouse_move", "x": 100, "y": 200})
viewer.disconnect()
Narrow who may connect at all with an IP allow-list (CIDR ranges or exact addresses); peers outside it are rejected during the handshake:
RemoteDesktopHost(token="tok", ip_allowlist=["10.0.0.0/8", "192.168.1.100"])
Platform support
| Platform | Backend | Input | Screen capture | Recording | Window management |
|---|---|---|---|---|---|
| Windows 10 / 11 | Win32 ctypes (+ optional Interception driver) | ✅ | ✅ | ✅ | ✅ |
| macOS 10.15+ | pyobjc / Quartz | ✅ | ✅ | ❌ | ❌ |
| Linux X11 | python-Xlib (+ optional uinput) |
✅ | ✅ | ✅ | ❌ |
| Linux Wayland | libei, or ydotool / wtype / grim | ✅ | ✅ | ❌ | ❌ |
| Android | adb + uiautomator2 | ✅ | ✅ | — | — |
| iOS | WebDriverAgent / facebook-wda | ✅ | ✅ | — | — |
Wayland forbids global input recording for unprivileged clients — set
JE_AUTOCONTROL_LINUX_DISPLAY_SERVER=x11 to record on an X11 session. Window
management is currently Windows-only and raises a clear NotImplementedError
elsewhere. Opt-in driver-level backends (JE_AUTOCONTROL_WIN32_BACKEND=interception,
JE_AUTOCONTROL_LINUX_BACKEND=uinput, ViGEm virtual gamepad) exist for apps that
ignore synthetic input, and fall back silently when the driver is absent.
Documentation and examples
| Resource | What's in it |
|---|---|
examples/ |
27 self-contained scripts: screenshot + click, OCR, scheduler, remote desktop, agent loop, observability, recording, variables, hotkeys, triggers, reports, MCP, REST, secrets, plugins, computer use, Wayland, cross-host DAGs, chat-ops, pytest/BDD, anchor locators. |
| Read the Docs | Full API reference, English and 中文. |
| architecture_explore.md | Every module's responsibility, layer by layer. |
| docs/CAPABILITY_MATRIX.md | Capability × platform matrix. |
| docs/API_LIFECYCLE.md | Stable-API and deprecation policy. |
| WHATS_NEW.md | Per-release notes. |
| CHANGELOG.md | Compatibility changelog. |
| SECURITY.md | Security policy and reporting. |
Development
git clone https://github.com/Intergration-Automation-Testing/AutoControl.git
cd AutoControl
pip install -r dev_requirements.txt
uv sync # or: reproducible install from the committed uv.lock
python -m pytest test/unit_test/headless # headless unit tests
python -m pytest test/integrated_test/ # cross-module workflows
ruff check je_auto_control/
pylint je_auto_control/
bandit -c pyproject.toml -r je_auto_control/
Contributions are welcome — see CONTRIBUTING.md and
CODE_OF_CONDUCT.md. Two rules the CI enforces: import je_auto_control must never pull in PySide6, and every feature needs both a headless
API and a GUI surface.
License
MIT License © JE-Chen. See Third_Party_License.md for the licenses of bundled and optional third-party components.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file je_auto_control-0.0.218.tar.gz.
File metadata
- Download URL: je_auto_control-0.0.218.tar.gz
- Upload date:
- Size: 1.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
29b156b80ff69fed20a5b8b3b259cba899fa4e3e7ef0080ab37cf10de17b42dc
|
|
| MD5 |
9f742fa116e6a01711c6877bdeff48cb
|
|
| BLAKE2b-256 |
712bdd585971b3c40b55296f9f3e20924cd01275b042a5efb801c691ef7549b7
|
File details
Details for the file je_auto_control-0.0.218-py3-none-any.whl.
File metadata
- Download URL: je_auto_control-0.0.218-py3-none-any.whl
- Upload date:
- Size: 1.8 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
66bc8ae957118900599b6344312a8d40d89d5f19a686ba389d94f0dd4e82ed70
|
|
| MD5 |
8f2a7ee2efbc306983194d937141d17f
|
|
| BLAKE2b-256 |
5e8ea2cfa229fa6f11ab7ab4ba9941cd28a1a3fe9c161f833691b1b16aac6bf1
|
