PyClickerRPA

A visual UI automation & testing tool. Automate Windows apps and Android devices/emulators by what's on screen — image/text matching + mouse/keyboard (or ADB) input — then turn those flows into a visual test framework with assertions, visual-regression baselines, HTML/JUnit reports, and a web console for remote runs.

Python/PySide6, started as a port of a Qt/C++ AutoClicker. Legacy .json tasks remain supported; structured control-flow blocks are a Python-only extension.

Install

pip install PyClickerRPA

Optional extras:

pip install "PyClickerRPA[ocr]"   # PaddleOCR — find-text / read-text steps
pip install "PyClickerRPA[web]"   # FastAPI + uvicorn — web console

Entry points

autoclicker                          # desktop GUI (flow editor / list editor / test tab)
autoclicker-run task.json --window "Notepad"   # headless run against a window
autoclicker-run task.json --adb                # run against an Android device (adb)
autoclicker-lint task.json                     # health-check a task (exit code = error count)
autoclicker-testkit suite.json --report out.html --junit out.xml   # run a test suite
autoclicker-web --suites-dir examples          # web console at http://127.0.0.1:8000

Automation engine

• Targets: any visible window, the whole screen (multi-monitor + high-DPI aware), or an Android device/emulator via ADB (--adb [serial]).
• Image matching: OpenCV template match with self-healing by default — template → multi-scale → ORB feature match fallback (survives scaling / minor UI changes); transparent-mask templates (PNG alpha) ignore don't-care regions. Per-step matchMode = robust / template / feature.
• Also: pixel-color checks, optional PaddleOCR text find / read-to-variable.
• Smart wait: wait_stable waits until the screen (or an ROI) stops changing — replaces guessed sleeps (fast when ready, patient when slow).
• ~30+ step types: click image/text/position, click-all, relative/offset clicks, wait / random / wait-stable, drag / drag-from-image / long-press / pinch / scroll, key / type / hotkey, loop / goto / while / do-until, if image/text/color/ count/var with true/false branching (if-else), set-var, read-text→variable, activate-window, HTTP request, run-Python-script, call sub-task, groups, and a stop step to end a run — with retry / repeat / until policies.

Black-box automation runtime

• Recognition-driven dispatch (MAA-style): a step's next list is scanned in order and the first candidate whose recognition gate hits is executed. Set next_timeout_ms (>0, or -1 for unbounded) with rate_limit_ms to wait and rescan instead of failing immediately — next_timeout_ms: 0 (default) keeps the classic single-pass behavior.
• Resource bundles: Task.resource_paths lists bundle root directories; template names that aren't real paths resolve as <root>/image/<name> with later roots overriding earlier ones (overlay). Task.load_bundle(dir) assembles a task from interface.json (resource roots + pipeline entry) or a plain task.json. Hot reload rides the signature-checked template cache.
• Configurable capture/input channels: controllers accept screencap_method / input_method. Win32: gdi_bitblt (default), printwindow, input sendinput (default) or postmessage (background, needs a window handle; continuous gestures unsupported). ADB: screencap (default) or raw_gzip (device-side gzip, steadier on slow links), input adb_input. minicap / minitouch / maatouch / desktop_dup are registered placeholders that fall back to the default with a strategy_fallback warning — they require extra device components that are not bundled (real-device behavior unverified).
• Target resolution (MAA-style): an action's target param can be a fixed region [x, y, w, h] (clicks its center) or a node name — it then clicks the center of that node's latest recognition box, enabling "recognize A, then click next to A". target: true / absent keeps the step's own targeting.
• Post-action freeze wait: post_wait_freezes: {time, target} on any action waits until the watched region stops changing before moving on — the standard remedy for loading screens and animations. Unlike a standalone wait_stable step it never fails the run; it waits up to time and proceeds.
• Human-like jitter: click coordinates are sampled from a truncated Gaussian (sigma = randomOffset/2, clamped to the bound) instead of a uniform spread, which looks more like a real hand and is friendlier to anti-automation checks.

Authoring (desktop GUI)

Drag-and-drop node flow graph and an interoperable step list, with nested If/Else, While, fixed-count loops, ForEach arrays, and Break/Continue — no step numbers required. Double-click a block to edit its body or branches. Legacy condition nodes retain their true/false ports.

Select consecutive steps and choose 包进… → 遍历数组 to reuse them for each item. Enter a custom JSON array or generate a numeric range, then reference the current item in text actions, assignments, and conditions. For example, iterate [1,2,3,4,5,6,7,8,9,10] with variable server, click {server}服, perform the actions, and return to the server-selection screen inside the same body. Enable exact text matching so 1服 does not also match 11服. A failed action inside a block stops execution unless its failure policy explicitly skips it.

Also includes a snippet library, task packages (.zip with nested templates), on-preview coordinate/color picking, template crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler, and an image/color monitor/trigger with tray icon; a Test tab to load and run visual test suites.

Smart recorder

The action recorder (global hooks) doesn't just replay coordinates and sleeps — with 智能录制 / smart mode on (default) it records what things look like instead of where they were:

• Clicks become image steps. A patch is cropped around the click and grown (48 → 192 px) until it is both textured enough and unique on screen, then emitted as a click_image step — so the flow still works after the window moves or the resolution changes. Identical patches are de-duplicated by content hash.
• Drags become drag_image. The drag's start is anchored to an image and the displacement is stored, so start and end move together.
• Pauses become wait_stable. A long pause is only converted if the recorder actually observed the screen settle during it; on a permanently animating screen it keeps the fixed wait (a wait_stable that never settles would fail the run).

Anything that can't be made robust falls back to the plain coordinate/fixed-wait step and says why in the log.

Diagnostics

Task health check (autoclicker-lint task.json, or 诊断 → 任务体检 in the GUI) statically scans a task and separates will definitely break from works but fragile:

• errors — missing template / sub-task file, image step with no template, loop / goto / fail-jump target out of range, unbalanced groups. The GUI asks for confirmation before running a task with errors; the CLI exit code is the error count.
• warnings — steps that depend on absolute coordinates, long fixed waits, a match threshold low enough to hit the wrong thing, templates too small to be unique. These are exactly the things the smart recorder now avoids, so the check doubles as a "what should I re-record?" list.

Failure attribution. A run captures a screenshot per step; the first successful run becomes the baseline timeline (.timelines/<task>/). When a later run fails, the frames are compared step by step and the report names the step where the screen first diverged from the successful run — which is usually earlier than the step that failed (step 7 can't find its button because a popup appeared at step 4). You get the diverging step, the per-step changed-pixel ratios, and a baseline|actual|heatmap image. If the screens match, it says so — that points at timing/flakiness rather than a UI change.

Visual test framework (autoclicker.testkit)

Turn automation into tests:

• Assertions: image / text / color present-or-absent, variable comparison.
• Visual regression: baseline diff (first run captures the baseline; later runs flag changed-pixel ratio and emit a baseline|actual|heatmap image).
• Cases & suites: declarative JSON suites or pytest fixtures (screen, visual); data-driven cases (a data table / CSV runs the case per row).
• Reports: self-contained HTML (embedded screenshots + per-step timeline, optional GIF) and JUnit XML for CI; retries mark flaky.

# pytest style
def test_login(screen, visual):
    screen.assert_image("assets/login_btn.png")
    screen.assert_text("Username")
    visual.check("login.png", tolerance=0.02)

Web console (autoclicker-web)

Browser UI to trigger suites (single or batched), watch live per-line logs and per-case results, schedule recurring runs, list ADB devices and grab screenshots, browse persisted run history, and open the HTML report — all served by FastAPI with a zero-build vanilla-JS front end.

Notes

• Host is Windows (Win32 capture + SendInput); ADB targets are cross-platform via the adb CLI (must be on PATH).
• OCR's first run downloads PaddleOCR models (~100 MB).
• Some target windows may require running as administrator.

License

MIT