PyClickerRPA

A visual UI automation & testing tool. Automate Windows apps and Android devices/emulators by what's on screen — image/text matching + mouse/keyboard (or ADB) input — then turn those flows into a visual test framework with assertions, visual-regression baselines, HTML/JUnit reports, and a web console for remote runs.

Python/PySide6, started as a port of a Qt/C++ AutoClicker (still reads/writes the same .json task format).

Install

pip install PyClickerRPA

Optional extras:

pip install "PyClickerRPA[ocr]"   # PaddleOCR — find-text / read-text steps
pip install "PyClickerRPA[web]"   # FastAPI + uvicorn — web console

Entry points

autoclicker                          # desktop GUI (flow editor / list editor / test tab)
autoclicker-run task.json --window "Notepad"   # headless run against a window
autoclicker-run task.json --adb                # run against an Android device (adb)
autoclicker-testkit suite.json --report out.html --junit out.xml   # run a test suite
autoclicker-web --suites-dir examples          # web console at http://127.0.0.1:8000

Automation engine

• Targets: any visible window, the whole screen (multi-monitor + high-DPI aware), or an Android device/emulator via ADB (--adb [serial]).
• Image matching: OpenCV template match with self-healing by default — template → multi-scale → ORB feature match fallback (survives scaling / minor UI changes); transparent-mask templates (PNG alpha) ignore don't-care regions. Per-step matchMode = robust / template / feature.
• Also: pixel-color checks, optional PaddleOCR text find / read-to-variable.
• Smart wait: wait_stable waits until the screen (or an ROI) stops changing — replaces guessed sleeps (fast when ready, patient when slow).
• ~30+ step types: click image/text/position, click-all, relative/offset clicks, wait / random / wait-stable, drag / long-press / pinch / scroll, key / type / hotkey, loop / goto / while / do-until, if image/text/color/ count/var with true/false branching (if-else), set-var, read-text→variable, activate-window, HTTP request, run-Python-script, call sub-task, groups, and a stop step to end a run — with retry / repeat / until policies.

Authoring (desktop GUI)

Drag-and-drop node flow graph (condition nodes have true/false branch ports for visual if-else; a terminal stop node ends the flow) and an interoperable step list with a full per-step dialog; an action recorder (global hooks), snippet library, task packages (.zip with templates), on-preview coordinate/color picking, template crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler, and an image/color monitor/trigger with tray icon; a Test tab to load and run visual test suites.

Visual test framework (autoclicker.testkit)

Turn automation into tests:

• Assertions: image / text / color present-or-absent, variable comparison.
• Visual regression: baseline diff (first run captures the baseline; later runs flag changed-pixel ratio and emit a baseline|actual|heatmap image).
• Cases & suites: declarative JSON suites or pytest fixtures (screen, visual); data-driven cases (a data table / CSV runs the case per row).
• Reports: self-contained HTML (embedded screenshots + per-step timeline, optional GIF) and JUnit XML for CI; retries mark flaky.

# pytest style
def test_login(screen, visual):
    screen.assert_image("assets/login_btn.png")
    screen.assert_text("Username")
    visual.check("login.png", tolerance=0.02)

Web console (autoclicker-web)

Browser UI to trigger suites (single or batched), watch live per-line logs and per-case results, schedule recurring runs, list ADB devices and grab screenshots, browse persisted run history, and open the HTML report — all served by FastAPI with a zero-build vanilla-JS front end.

Notes

• Host is Windows (Win32 capture + SendInput); ADB targets are cross-platform via the adb CLI (must be on PATH).
• OCR's first run downloads PaddleOCR models (~100 MB).
• Some target windows may require running as administrator.

License

MIT