AutoControl

AutoControl is an open-source, cross-platform computer-use and GUI automation framework for AI agents, Python applications, and automated testing. It drives the mouse and keyboard, finds UI targets through template matching, OCR, the OS accessibility tree, or a vision model, records and replays flows, and runs deterministic JSON actions across Windows, macOS, Linux (X11 and Wayland), the BSDs, Android, and iOS.

Every capability ships three ways: a Python API, an AC_* action command usable from JSON files / CLI / servers, and a GUI tab. Nothing is GUI-only.

繁體中文 · 简体中文

Why AutoControl

• One API, seven platforms. wrapper/platform_wrapper.py picks the backend at import time; your script does not change between Windows, macOS, X11, and Wayland.
• Scriptable without Python. 797 AC_* commands cover the whole feature set, so a JSON file can do anything the library can — including loops, branches, try/catch, macros, and variables.
• Headless by default. import je_auto_control never loads Qt. The GUI is an optional extra that wraps the same headless core.
• Locate things four ways. Template matching, OCR, the accessibility tree, and a vision-language model — composable through anchor locators and self-healing fallbacks.
• Light dependency floor. The REST server, JSON Schema validator, JWT, TOTP, WebSocket framing, ACME client, USB/IP protocol, and Prometheus metrics are all standard-library implementations. Heavy things are opt-in extras.

AI agents and computer use

AutoControl is an open-source, cross-platform computer-use and GUI automation framework for AI agents.

Use it when an agent needs to operate a real desktop GUI — native applications, real browser windows, remote desktops, or mobile devices — rather than only a browser DOM/API.

When an AI agent should choose AutoControl

• Real mouse and keyboard input
• Screenshots and screen-coordinate interaction
• OCR and native accessibility-tree discovery
• Image/template matching
• Vision-language-model UI localization
• Anchor and self-healing locators
• Cross-platform desktop/mobile automation
• Deterministic JSON action files
• MCP-based computer use

A robust agent loop is:

observe → identify → act → wait → verify → recover

The repository also has a dedicated AI agent guide with MCP configuration, tool selection, OpenAI integration, security guidance, and searchable computer-use terminology.

AI-friendly MCP

Start the stdio server with:

pip install je_auto_control
je_auto_control_mcp

The MCP server exposes the canonical ac_* surface plus short, model-friendly aliases such as click, type, screenshot, find_text, click_text, drag, focus_window, and wait_image.

For inspection-only clients:

je_auto_control_mcp --read-only

If a client needs only canonical ac_* names:

JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp

For OpenAI agent integrations, expose a focused allow-list with export_openai_tools(only=[...]) instead of passing the complete AutoControl command catalogue. This both fits provider limits and reduces the authority given to the model.

Project identity

Project: AutoControl
Repository: Integration-Automation/AutoControlGUI
Python package / PyPI: je_auto_control
MCP command: je_auto_control_mcp

Choosing the right automation layer

AutoControl is not intended to replace every automation tool. Use the smallest layer that matches the interface:

The differentiator is the combination of real computer input + semantic/visual discovery + self-healing + agent/MCP integration behind one cross-platform automation surface.

Installation

pip install je_auto_control            # core
pip install je_auto_control[gui]       # + PySide6 desktop app

Optional extras, installed only when you need them:

Closing a factory-built feature with the tab cross releases its widget, subscriptions and timers on the next Qt event-loop turn; reopening starts a new panel. View → Tabs and the embedding hide_tab API only hide it and preserve state. tab_registry.open/close and close_tab expose explicit lifecycle operations. Legacy owner-built forms and Remote Desktop remain reusable on close while their lifecycle migration is pending.

Windows on arm64 installs and runs, minus what upstream cannot ship there: neither opencv-python nor cryptography publishes a win_arm64 wheel. So find_image*, screenshot() (the OpenCV/BGR one — the Pillow capture still works), the secret vault, action-file encryption, ACME/TLS and encrypted recording each raise a message naming the missing wheel instead of failing obscurely. Mouse, keyboard, screen size, window management, the accessibility tree, the action executor, the MCP/REST/TCP servers and the GUI all work — measured, not assumed. Every other platform is unaffected.

Requirements: Python ≥ 3.10 (≥ 3.11 on Windows arm64, which is where CPython's official builds for it start). On Linux, install build prerequisites first:

sudo apt-get install cmake libssl-dev

OCR, VLM, and LLM backends (pytesseract, easyocr, paddleocr, anthropic, openai) are loaded on demand — install whichever you actually use. For Tesseract, find_tesseract_cmd() locates the executable ($TESSERACT_CMD, then PATH, then the installers' default folders), set_tessdata_dir() points it at a language-data folder, and ocr_status() / ocr_languages() say what is missing before the first OCR call.

Log file: the library logs to ~/.je_auto_control/logs/AutoControlGUI.log, created on the first record (importing alone writes nothing) and shared by every process on the account (appended to, one process id per line, moved to .1 past 10 MB). Set JE_AUTOCONTROL_LOG_FILE to write elsewhere, or to os.devnull to turn the file off.

60-second quick start

1. As a Python library

import je_auto_control as ac

ac.set_mouse_position(500, 300)
ac.click_mouse("mouse_left")
ac.write("Hello World")
ac.hotkey(["ctrl_l", "s"])

x, y = ac.locate_image_center("save_button.png", detect_threshold=0.9)
ac.click_text("Submit")                       # OCR
ac.click_accessibility_element(name="OK")     # accessibility tree
ac.click_by_description("the green Submit button")   # vision model
ac.screenshot("shot.png", screen_region=[0, 0, 800, 600])

2. As a JSON action file — flow.json

[
    ["AC_set_var", {"name": "user", "value": "alice"}],
    ["AC_locate_and_click", {"image": "login.png", "mouse_keycode": "mouse_left"}],
    ["AC_write", {"write_string": "${user}"}],
    ["AC_retry", {"max_attempts": 3, "body": [
        ["AC_wait_text", {"target": "Welcome", "timeout": 10}]
    ]}],
    ["AC_assert_text", {"text": "Welcome"}],
    ["AC_generate_html_report", {"html_name": "report"}]
]

je_auto_control run flow.json --var user=bob
je_auto_control run flow.json --dry-run     # list the steps without touching the mouse

3. As a desktop app

pip install je_auto_control[gui]
python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()"

Record a flow, edit it in the visual Script Builder, and save it as the same JSON format the CLI runs. (python -m je_auto_control is the legacy action-file runner — -e, -d, -c, --execute_str — not the GUI.)

Capability overview

Every row works headlessly. "GUI tab" is where the same feature surfaces in the desktop app; tab commands live in the window's Actions menu.

Self-healing propagates cancellation before fallback or clicking. Its audit log masks secrets already resolved in the current run before writing or returning records; returned locator outcomes retain their original values. Legacy log files are not rewritten.

Live image/VLM self-healing now shares one lazy logical capture, including its actual clipped origin. screen_region is [left, top, right, bottom] for both strategies. Schema-v2 audit records add run/step identity, content/configuration version, frame hash, strategy/capture timing and resolved backend/model; old records still load. Click dispatch without a verifier is recorded as issued or failed, with verification left unknown. The GUI exposes region input and selected-record evidence; Script Builder keeps region JSON structured. Historical healed/heal_rate count located candidates; separate operation metrics require explicit verification.

Experimental self_heal_run(SelfHealRequest(...)) defaults to locating. Choose operation="click" and VerificationRequest(verifier=...) (a Python bool callback) or VerificationRequest(description=...) (a configured VLM) to check the post-condition after input dispatch. VLM verification obtains a fresh logical frame and may incur API cost. verified is true, false or unknown; found and issued alone do not verify the operation. Exceptions and cancellation propagate without retrying the click. AC_self_heal_run and MCP ac_self_heal_run accept a nested request; explicit false verification fails the command/CLI or returns MCP isError with the structured result. GUI Self-Healing offers an optional post-condition field, and Script Builder edits the JSON request. Standalone VLM capture now uses the same logical origin/Retina normalization in memory. Framework assertions stop lookup recovery.

Captured self-healing rows now include an optional journal_path alongside run/step IDs. Experimental inspect_heal_source(path, run_id, step_id), AC_self_heal_source and MCP ac_self_heal_source read only the original step’s sanitized lifecycle snapshots. In Self-Healing, select a row and choose Actions → Inspect original step to open the background reader. Switching targets waits for the older reader to stop and retains the requester’s context; closing cancels pending work. Run History’s reader also accepts an optional step ID. Run/step filtering precedes the output limit, the whole file is validated, current path roots are rechecked around progress callbacks, and cancellation returns no partial result. Legacy or uncaptured rows have no original-source link; inspection never replays a command.

Experimental local causal primitives provide SyncEntry, merge_entries, SyncPeers and SyncOutbox. Concurrent/forked versions retain both values; explicit resolution observes both clocks. Serialized peer acknowledgements gate collection eligibility and retired epochs require full resync. SQLite queues isolate endpoint/account identity, freeze prepared requests and retain bounded cancellable retries across restart. AC_sync_entry_merge, AC_sync_peer_state, AC_sync_outbox_enqueue, AC_sync_outbox_status and matching MCP/Builder Data commands expose review and local persistence. Pure transforms are observable; private queues require DRIVE. These primitives do not yet replace the HTTP client merge or wire server collection/full resync. See the causal/outbox guide and python -m examples.causal_sync.

Experimental persistent configuration uses ConfigStore.commit(..., base_revision=..., operation_id=...) and a version-two signaling PUT envelope. SQLite transactions preserve buckets and retry receipts across restart; stale revisions or changed operation identities conflict. ConfigSyncClient defaults to conditional writes; blind legacy writes require explicit client/server opt-in. --config-store selects the lazy database. Private AC_config_store_get / AC_config_store_commit and matching MCP tools share explicit Script Builder Data commands. This stores data without running scripts or enabling triggers; causal merge and offline outbox remain separate. See the persistent config guide and python -m examples.config_store.

Experimental journal-to-code candidates use generate_candidate_from_log(path, run_id=...), AC_generate_candidate_from_log, and MCP ac_generate_candidate_from_log. They reconcile lifecycle snapshots, retain recorded control sources and retry evidence, and mark partial reconstructions as observed paths. Failed/incomplete invocations and unknown or redacted inputs remain explicit exclusions; parallel order is never invented. Generated Python/pytest/Robot source passes static, command and non-dispatching dry-run checks. codegen --from-log run.jsonl --run-id selected prints a JSON review bundle; -o review.zip or write_candidate_archive atomically exports code, actions and provenance together. Actions → Review journal candidate in Recording Editor or Script Builder opens source/diff/manifest/warnings; Builder import is separate and refuses intervening editor changes. Neither generation nor import runs the candidate. See the journal candidate guide and python -m examples.journal_candidates.

Experimental template-candidate revisions use create_template_candidate, preview_template_candidate, verify_template_candidate, accept_template_candidate and revert_template_candidate, with matching AC_template_candidate_* / ac_template_candidate_* operations. A new immutable JSON bundle retains original bytes and the candidate encoded in the target’s PNG/JPEG/BMP format. Preview returns bounded thumbnails. Verification compares both frozen versions on a local sample manifest; acceptance requires a labelled hit, an expected miss and every labelled case correct, plus the reviewed bundle SHA-256. Acceptance rechecks the dataset and atomically replaces only matching original bytes; revert restores exact original bytes and refuses later edits. Unknowns and native-operation success remain separate. The recorded threshold applies to verification; activating image bytes does not edit script thresholds. Self-Healing → Template candidates offers all five Actions, background work, cancellation and JSON evidence. Run python -m examples.template_candidates for a temporary-file-only example.

Experimental fixed-frame comparisons use evaluate_locators or evaluate_healing_manifest, AC_self_heal_evaluate, MCP ac_self_heal_evaluate, and Self-Healing → Version comparison. Reports separate labelled accuracy, wrong predictions, unknowns and verified recovery, include p50/p95 and provenance, and export JSON/HTML through write_healing_report or GUI Actions. The local manifest operation never captures, clicks or selects a network backend. See the dataset, populations and limits.

Beyond this table, utils/ holds 311 headless packages covering assertions, resilience, data quality, i18n auditing, redaction, governance, observability, and more. The full per-module map is in architecture_explore.md.

Experimental structured journal capture is exported from je_auto_control (also available in utils.action_journal): journal_scope captures existing executor calls, and execute_action_with_journal isolates one run. Local loop/parallel/DAG steps retain parent IDs; failures, cancellation and dry-runs are distinct. Existing history owners automatically link explicit captures, including captures opened by local parallel/DAG workers; HistoryStore.journals_for_run returns the non-owning links. Clearing/pruning history leaves journal files intact. The CLI supports run script.json --journal run.jsonl [--journal-run-id ID] and journal run.jsonl [--run-id ID] [--limit 100]. Journal reads show sanitized lifecycle snapshots in append order; output defaults to 100 snapshots (maximum 10,000). AC_journal_capture accepts named parameters and runs a nested body on the owning executor without replacing its variables; AC_journal_read shares sanitized, bounded inspection with the CLI and public inspect_journal. Both commands are editable in Script Builder. Capture rejects identifiers or destinations containing secrets known to the run before history linking, file creation or action effects. MCP ac_journal_capture and ac_journal_read provide structured results, progress and cooperative cancellation. Captured journal roots and observers survive nested captures and local workers; reads can cancel between validated records. JournalRunOptions and execute_journal_run expose the same controls in Python.

Manual runs can opt into history with manual_history_scope, AC_history_capture, CLI run --history [--history-db PATH], or Record in Run History in Script Executor / Script Builder. An optional GUI journal path also works independently of history. GUI/CLI result JSON and existing variable scopes are preserved; the command returns history_id, record and failed_actions. Existing and nested captures link to the new manual row. CLI previews use the cli.preview source label; ordinary runs use cli.run.

Run History includes a read-only journal browser. Use Actions → Use Journals from Selected Run to select a provenance link, or Choose Journal for an external file. Set a run ID and snapshot limit, then Read Journal; start/completion snapshots stay separate and their full sanitized data appears on selection. Cancel Journal Read and closing the view request cancellation between records. File parsing runs in a background worker; selecting a link opens no file.

Values resolved from ${secrets.NAME} are masked in application diagnostics during executor runs, including nested actions and local parallel/DAG workers. CLI/REST/MCP requests and scheduled, trigger, hotkey and email runs retain the mask through error handling and audit/history writes; report records capture redacted snapshots. DAG calls retain the owner mask through complete node diagnostics, including standalone calls and custom local runners. Watchdog, observer and callback-executor calls also retain it through final diagnostics; independent rules have separate masks. CLI stderr is masked, while API outcomes, raised exceptions and CLI result payloads retain their original values. Masks are discarded when the owning request or run exits. Interrupted history runs are recorded as error, while journal steps remain cancelled; no error screenshot is taken. Cancellation propagates through non-strict execution, recovery/retry, callbacks and local workers. Hard exits retain their original signal even if history completion fails.

Command-line interface

Each je_auto_control run invocation, including --dry-run and runs without --var, gets a fresh variable scope. Top-level run_from_description and StateMachine.run() also isolate variables while sharing them across their own actions and states; nested calls reuse an existing execution scope. A caller-owned Executor() keeps its own variables.

je_auto_control run script.json [--var name=value] [--dry-run] [--allow-package NAME]
je_auto_control validate script.json          # alias: lint
je_auto_control fmt script.json [--check]
je_auto_control list-commands [--filter mouse] [--json]
je_auto_control record out.json [--duration 5]
je_auto_control codegen script.json --target pytest -o test_flow.py
je_auto_control failure-bundle failure.zip --error "login timed out"
je_auto_control list-jobs
je_auto_control start-server --port 9938      # TCP socket server
je_auto_control start-rest   --port 9939      # REST API
je_auto_control version

--var name=value is parsed as JSON when possible (count=10 becomes an int), otherwise kept as a string. run exits 1 when any action failed (the run still goes on to the end), so a CI step fails with it. The legacy python -m je_auto_control -e file.json entry point still works, and its -e, -d and --execute_str flags exit 1 the same way. Under TestPioneer, which sets TEST_PIONEER_ARTIFACT_DIR, the HTML, JSON and XML reports asked for under a relative name are written below that directory.

Servers and integrations

All servers bind to 127.0.0.1 unless you opt in explicitly.

The MCP window-state tools ac_window_minimize, ac_window_maximize and ac_window_restore return the window handle on accepted requests and isError: true on backend refusal (including a stale handle or rejected activation). Raw Win32 ShowWindow previous visibility is normalized by the backend.

Package gate. AC_add_package_to_executor and AC_add_package_to_callback_executor import a Python package and register its members as commands, so an action list arriving over any of these surfaces could load os or subprocess. No package loads unless it has been allowed: a package that is not on the allowlist is refused before it is imported, and that action fails with AutoControlExecuteActionException. Allow packages (submodules included) from Python with executor.allow_packages("name", …), for every entry point — both CLIs, the socket / REST / MCP servers and the scheduler — with the JE_AUTOCONTROL_ALLOWED_PACKAGES environment variable (comma-separated names, read when the process starts), or for one CLI run with je_auto_control run script.json --allow-package NAME (repeatable). executor.set_allow_arbitrary_packages(True) opens the gate for every package, which is what earlier releases did by default (with a DeprecationWarning). None of these is an AC_* command, so an action list cannot open its own gate.

Opt-in hardening. Each of these is off until configured, and a server without them behaves as before. JE_AUTOCONTROL_RBAC_USERS=<user store file> makes the REST API and the MCP HTTP transport resolve the bearer token to a user and authorise each route, tool and privileged AC_* command by role (viewer / operator / admin); the shared token is then refused. JE_AUTOCONTROL_MCP_PATH_ROOTS (directories separated by the OS path separator) confines every MCP tool argument that is a file path to those directories, JE_AUTOCONTROL_MCP_PATH_ROOTS_FROM_CLIENT=1 adds the client's roots/list, and JE_AUTOCONTROL_MCP_ENV_REF_ALLOW limits which env:// names ac_resolve_ref may read. JE_AUTOCONTROL_ACTION_SIGNING_PUBLIC_KEY makes an endpoint verify Ed25519-signed action files without being able to sign them (create_signing_keypair; the private key stays on the signing machine). A remote-desktop viewer writes files pushed by a host only below ~/Downloads/AutoControl (JE_AUTOCONTROL_REMOTE_DOWNLOAD_DIR); that one is on by default.

The REST GUI shows RBAC mode and disables the shared-token field while a user store is active. rest_api_registry.status() includes rbac_enabled and returns token: null in that mode; use an individual user token. Custom REST gates that only implement check() remain supported in shared-token mode; RBAC requires an authenticated user context and refuses an identityless success.

RBAC viewers do not receive the named sensitive-read tools, even when their MCP readOnlyHint is true: clipboard/history, local document/data/SQL and dotenv reads, reference resolution, credential generation, persistent store readers, outbound checks, microphone capture and external VLM requests require operator/admin. Assertion containers also require operator because their specs can read those resources. Screen observation and pure input parsing keep their existing classification. ac_realize_item, ac_idempotency_begin and ac_rate_limit change state and are excluded from --read-only mode.

Delayed interval/cron jobs, triggers, hotkeys, watchdog rules, webhooks, e-mail triggers and observer callbacks retain their registrant’s identity. Store-backed users and roles are checked again on dispatch; deleted users, unavailable stores and viewers cannot execute deferred work. Starting a daemon does not transfer its registrations to the starter. Ownership metadata retains no RBAC bearer token; explicit Python AuthorizationContext values without store_path keep their captured role, and work registered outside RBAC clears worker identity. Observer/watchdog callbacks get fresh variable scopes, and failed watchdog actions do not count as dismissed popups. Nested callbacks retain the caller’s failure count; a webhook with a revoked owner returns HTTP 403 and fired: false.

How the remote-desktop wire protocol works

Worth knowing before you expose a host, and described nowhere else in the docs. The default transport is length-prefixed framing over raw TCP — no extra dependencies — and it opens with an HMAC-SHA256 challenge/response handshake: a viewer that fails auth is dropped before it is sent a single frame. JPEG frames are encoded at the configured FPS and quality and handed to authenticated viewers through a shared latest-frame slot, so a slow viewer drops frames instead of stalling the rest. Viewer input arrives as JSON and is validated against an allow-list of actions before being applied through the ordinary input wrappers, so a viewer cannot invent new operations.

# Be remoted — start a host and hand the token + port to whoever views you
from je_auto_control import RemoteDesktopHost
host = RemoteDesktopHost(token="hunter2", bind="127.0.0.1",
                         port=0, fps=10, quality=70)
host.start()
print("listening on", host.port, "viewers:", host.connected_clients)

# Control another machine — connect a viewer and send input
from je_auto_control import RemoteDesktopViewer
viewer = RemoteDesktopViewer(host="10.0.0.5", port=51234, token="hunter2",
                             on_frame=lambda jpeg: ...)
viewer.connect()
viewer.send_input({"action": "mouse_move", "x": 100, "y": 200})
viewer.disconnect()

Narrow who may connect at all with an IP allow-list (CIDR ranges or exact addresses); peers outside it are rejected during the handshake:

RemoteDesktopHost(token="tok", ip_allowlist=["10.0.0.0/8", "192.168.1.100"])

Platform support

¹ macOS recording captures through a Quartz event tap and needs Accessibility permission (System Settings → Privacy & Security → Accessibility). Without it recording raises and names the permission rather than returning an empty session.

² The BSDs run the X11 backend unchanged — the same X server, the same python-Xlib, which is the only dependency input, recording and window management have. A freebsd CI job drives real input on a real FreeBSD 14 and reads it back off the X server; OpenBSD and NetBSD take the same code path but have no CI runner. Screen capture is the exception, and the reason is packaging rather than the platform: it goes through Pillow/mss and OpenCV, and opencv-python, pillow and cryptography publish no FreeBSD wheels. Build those from ports and capture, image matching, OCR and action encryption work too — import je_auto_control no longer requires any of them.

Wayland input falls back to the ydotool CLI wherever libei is not reachable, and that fallback needs ydotool 1.0 or newer. Every argument AutoControl builds arrived in that release; 0.1.x — which is what Debian bookworm and every current Ubuntu still ship under that name, and Debian trixie ships not at all — answers the same arguments with exit code 0 and no events. AutoControl detects it and refuses rather than reporting success for input it never sent. Arch, Fedora and Debian unstable package 1.0.

That fallback also positions the pointer accurately only where the compositor's pointer acceleration is off. ydotool mousemove --absolute sends no absolute event: it drives the cursor into the corner the compositor clamps to and then moves relative to it, so the compositor accelerates the move — measured against a real wlroots session, libinput's default profile travels exactly twice the distance asked for. ydotool's own --help says the same; AutoControl logs it once per process rather than mispositioning in silence. Turn acceleration off for the ydotoold device (sway: input type:pointer accel_profile flat and pointer_accel 0), or install liboeffis so the libei path — absolute at the protocol level — is used.

The factor is the compositor's own setting and no client can read it back, so only you know whether it is off. JE_AUTOCONTROL_WAYLAND_POINTER_ACCEL=flat says it is, and moves silently; =strict refuses the move rather than let a click land somewhere else; leaving it unset keeps the warn-and-move default.

Wayland screen capture needs the tool your compositor supports, because no single one covers them all: grim on wlroots compositors (sway, Hyprland, river), gnome-screenshot on GNOME, spectacle on KDE. Install one and every capture path — screenshots, image and anchor locators, OCR, screen recording, remote desktop — goes through it. With none of them installed, gdbus is enough: xdg-desktop-portal is tried last, though it may ask for consent the first time. Failing that, capture fails loudly with an install hint rather than returning the blank XWayland root. The screen_capture check in je_auto_control.api.run_diagnostics() (and the GUI Diagnostics tab) reports which tier is in use.

One Wayland-only difference to plan around: a capture may contain the mouse cursor. Nothing here asks for it, but wlroots composites a software cursor into the output buffer whenever the backend has no cursor plane — which includes any session run with WLR_NO_HARDWARE_CURSORS=1 — and that buffer is what screen capture hands back. Windows and X11 never include the pointer, so a locator, a template match or an OCR read can find a pointer-shaped hole in the middle of its target here and nowhere else. Wayland does not let a client read the cursor position, so there is nothing to reliably mask or move around it: park the pointer away from what you are about to capture. The screen_capture check reports this as cursor_may_be_captured.

For a setup none of that fits, name your own command — it wins over every detected tool, and {output} is replaced with a temporary PNG path:

export JE_AUTOCONTROL_WAYLAND_CAPTURE_COMMAND="mycapture --png {output}"

Wayland forbids global input recording for unprivileged clients — set JE_AUTOCONTROL_LINUX_DISPLAY_SERVER=x11 to record on an X11 session. Window management works on Windows, macOS (pyobjc) and X11, including XWayland; on a pure Wayland session, whose protocol hides other clients' windows, list_windows() returns an empty list and every window action raises AutoControlUnsupportedOperationException saying why. Opt-in driver-level backends (JE_AUTOCONTROL_WIN32_BACKEND=interception, JE_AUTOCONTROL_LINUX_BACKEND=uinput, ViGEm virtual gamepad) exist for apps that ignore synthetic input, and fall back silently when the driver is absent.

Documentation and examples

Development

git clone https://github.com/Integration-Automation/AutoControlGUI.git
cd AutoControl
pip install -r dev_requirements.txt
uv sync                 # or: reproducible install from the committed uv.lock

python -m pytest test/unit_test/headless      # headless unit tests
python -m pytest test/integrated_test/        # cross-module workflows

ruff check je_auto_control/
pylint je_auto_control/
bandit -c pyproject.toml -r je_auto_control/

Contributions are welcome — see CONTRIBUTING.md and CODE_OF_CONDUCT.md. Two rules the CI enforces: import je_auto_control must never pull in PySide6, and every feature needs both a headless API and a GUI surface.

License

MIT License © JE-Chen. See Third_Party_License.md for the licenses of bundled and optional third-party components.

• Homepage: https://github.com/Integration-Automation/AutoControlGUI
• PyPI: https://pypi.org/project/je_auto_control/
• Documentation: https://autocontrol.readthedocs.io/en/latest/