PyClickerRPA

A visual UI automation & testing tool. Automate Windows apps and Android devices/emulators by what's on screen — image/text matching + mouse/keyboard (or ADB) input — then turn those flows into a visual test framework with assertions, visual-regression baselines, HTML/JUnit reports, and a web console for remote runs.

Python/PySide6, started as a port of a Qt/C++ AutoClicker (still reads/writes the same .json task format).

Install

pip install PyClickerRPA

Optional extras:

pip install "PyClickerRPA[ocr]"   # PaddleOCR — find-text / read-text steps
pip install "PyClickerRPA[web]"   # FastAPI + uvicorn — web console

Entry points

autoclicker                          # desktop GUI (flow editor / list editor / test tab)
autoclicker-run task.json --window "Notepad"   # headless run against a window
autoclicker-run task.json --adb                # run against an Android device (adb)
autoclicker-lint task.json                     # health-check a task (exit code = error count)
autoclicker-testkit suite.json --report out.html --junit out.xml   # run a test suite
autoclicker-web --suites-dir examples          # web console at http://127.0.0.1:8000

Automation engine

• Targets: any visible window, the whole screen (multi-monitor + high-DPI aware), or an Android device/emulator via ADB (--adb [serial]).
• Image matching: OpenCV template match with self-healing by default — template → multi-scale → ORB feature match fallback (survives scaling / minor UI changes); transparent-mask templates (PNG alpha) ignore don't-care regions. Per-step matchMode = robust / template / feature.
• Also: pixel-color checks, optional PaddleOCR text find / read-to-variable.
• Smart wait: wait_stable waits until the screen (or an ROI) stops changing — replaces guessed sleeps (fast when ready, patient when slow).
• ~30+ step types: click image/text/position, click-all, relative/offset clicks, wait / random / wait-stable, drag / drag-from-image / long-press / pinch / scroll, key / type / hotkey, loop / goto / while / do-until, if image/text/color/ count/var with true/false branching (if-else), set-var, read-text→variable, activate-window, HTTP request, run-Python-script, call sub-task, groups, and a stop step to end a run — with retry / repeat / until policies.

Authoring (desktop GUI)

Drag-and-drop node flow graph (condition nodes have true/false branch ports for visual if-else; a terminal stop node ends the flow) and an interoperable step list with a full per-step dialog; snippet library, task packages (.zip with templates), on-preview coordinate/color picking, template crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler, and an image/color monitor/trigger with tray icon; a Test tab to load and run visual test suites.

Smart recorder

The action recorder (global hooks) doesn't just replay coordinates and sleeps — with 智能录制 / smart mode on (default) it records what things look like instead of where they were:

• Clicks become image steps. A patch is cropped around the click and grown (48 → 192 px) until it is both textured enough and unique on screen, then emitted as a click_image step — so the flow still works after the window moves or the resolution changes. Identical patches are de-duplicated by content hash.
• Drags become drag_image. The drag's start is anchored to an image and the displacement is stored, so start and end move together.
• Pauses become wait_stable. A long pause is only converted if the recorder actually observed the screen settle during it; on a permanently animating screen it keeps the fixed wait (a wait_stable that never settles would fail the run).

Anything that can't be made robust falls back to the plain coordinate/fixed-wait step and says why in the log.

Diagnostics

Task health check (autoclicker-lint task.json, or 诊断 → 任务体检 in the GUI) statically scans a task and separates will definitely break from works but fragile:

• errors — missing template / sub-task file, image step with no template, loop / goto / fail-jump target out of range, unbalanced groups. The GUI asks for confirmation before running a task with errors; the CLI exit code is the error count.
• warnings — steps that depend on absolute coordinates, long fixed waits, a match threshold low enough to hit the wrong thing, templates too small to be unique. These are exactly the things the smart recorder now avoids, so the check doubles as a "what should I re-record?" list.

Failure attribution. A run captures a screenshot per step; the first successful run becomes the baseline timeline (.timelines/<task>/). When a later run fails, the frames are compared step by step and the report names the step where the screen first diverged from the successful run — which is usually earlier than the step that failed (step 7 can't find its button because a popup appeared at step 4). You get the diverging step, the per-step changed-pixel ratios, and a baseline|actual|heatmap image. If the screens match, it says so — that points at timing/flakiness rather than a UI change.

Visual test framework (autoclicker.testkit)

Turn automation into tests:

• Assertions: image / text / color present-or-absent, variable comparison.
• Visual regression: baseline diff (first run captures the baseline; later runs flag changed-pixel ratio and emit a baseline|actual|heatmap image).
• Cases & suites: declarative JSON suites or pytest fixtures (screen, visual); data-driven cases (a data table / CSV runs the case per row).
• Reports: self-contained HTML (embedded screenshots + per-step timeline, optional GIF) and JUnit XML for CI; retries mark flaky.

# pytest style
def test_login(screen, visual):
    screen.assert_image("assets/login_btn.png")
    screen.assert_text("Username")
    visual.check("login.png", tolerance=0.02)

Web console (autoclicker-web)

Browser UI to trigger suites (single or batched), watch live per-line logs and per-case results, schedule recurring runs, list ADB devices and grab screenshots, browse persisted run history, and open the HTML report — all served by FastAPI with a zero-build vanilla-JS front end.

Notes

• Host is Windows (Win32 capture + SendInput); ADB targets are cross-platform via the adb CLI (must be on PATH).
• OCR's first run downloads PaddleOCR models (~100 MB).
• Some target windows may require running as administrator.

License

MIT