pyguitest-recorder

Record desktop GUI activity and generate pyguitest scripts from it.

The point is not to replay a macro. It is to turn what you did into test code you would have been willing to write by hand — so a recorded click on a button becomes

gui.button("Save").click()

rather than gui.move_mouse(180, 90); gui.click(), which stops working the moment the window moves, the theme changes, or someone adds a toolbar item.

Quick start

pip install 'pyguitest-recorder[x11,atspi]'          # Linux, XWayland
pip install pyguitest-recorder 'pyguitest[windows]'  # Windows
pip install 'pyguitest-recorder[macos]'              # macOS

pyguitest-recorder --doctor              # can this machine record? why not?
pyguitest-recorder -o login_test.py      # record until Escape, Escape
python3 login_test.py                    # replay it

⚠️ Keyboard capture sees every application's keystrokes, not only the one you are recording — including your password manager. That is what X11's RECORD extension and Windows' low-level input hooks both do, and it is why this tool exists at all. Close what you would not want in a file, and read Privacy before recording anything that touches a login.

Recording needs X11, XWayland, or native Windows; under a Wayland session it reaches XWayland clients and nothing else, and says so rather than producing a file with silent gaps. Why that is permanent.

Three flags carry most of the value:

pyguitest-recorder -o login_test.py --save-session rec.json   # keep both
pyguitest-recorder --regenerate rec.json -o out.py            # re-render, no recording
pyguitest-recorder --record-motion                            # capture hovers too

Recording stops on Escape pressed twice, not only Ctrl-C — a recorder you can only stop from its own terminal is one you cannot stop while driving a full-screen application. Ctrl+1 records a check on whatever the pointer is over. Both are rebindable; see docs/recipes.md.

docs/getting-started.md walks through all of this properly.

What comes out

"""Generated by pyguitest-recorder. Edit freely.

Profile:     pyguitest-0.15
Recorded on: x11 (mutter)
Timeouts:    seconds; 3x the wait the recording observed (floor 10s, cap 120s)
"""

import pyguitest
from pyguitest import Capability, Role


def main() -> None:
    """Replay the recorded interaction."""
    with pyguitest.connect() as gui:
        gui.require(
            Capability.ELEMENT_ACTION,
            Capability.ELEMENT_TREE,
            Capability.WINDOW_ACTIVATE,
        )

        editor = gui.wait_for_window("Example", timeout=10)
        gui.activate_window(editor)
        gui.text_field("Name").set_text("Ada")
        gui.button("Save").click()


if __name__ == "__main__":
    main()

Plain pyguitest source, depending on nothing from this package. Three things about it are deliberate.

Elements lead, coordinates follow

Each click is resolved at record time against the accessibility tree, and falls down a ladder only as far as it has to:

    gui.button("Save").click()      a named element — survives redesigns,
                                    themes, resizes and added toolbar items
        ↓  nothing accessible under the pointer?
    window-relative coordinates     survives the window moving
        ↓  no window accounts for the point?
    gui.move_mouse(842, 612)        absolute — breaks when anything moves

--absolute-coordinates forces the bottom rung; --relative-coordinates forces the middle one.

The recorder refuses to name an element it cannot corroborate — if the element's process does not own the window under that point, or its own extents do not contain the point it was looked up at, the click becomes a coordinate instead. A coordinate that works beats a named element that does not. Every such refusal is written into the generated file's docstring, because "why is this script all coordinates?" is the first thing its reader asks. The rules, and the GTK4 measurements behind them, are in docs/developers/architecture.md; the fix when the answer is your own application is docs/testable-guis.md.

Typed text is the exception, and gets named anyway. A run of typing asks the toolkit what has focus rather than what is under the pointer — focus involves no geometry, so it still works where hit-testing has failed. A GTK4 application whose clicks all degrade to coordinates still produces gui.text_field("Name").set_text("Ada"), including for a field reached by Tab or focused by the application itself.

Scripts declare what they need

Every file opens with gui.require(...) naming the capabilities it uses, so a recording made on X11 and replayed somewhere weaker fails on the first line with a typed exception instead of halfway through with a click that went nowhere.

And the file is checked before it is offered. Generation compiles the script, confirms every gui.<method> call exists on the installed pyguitest.Session, and checks each Capability and Role constant against the same — because a recorder that emits a plausible script naming a function the library does not have is worse than no recorder.

Waits, not sleeps

The difference between a recorder and a macro player is what happens to the three seconds you spent waiting for a dialog. A macro player sleeps for three seconds: slow when the machine is fast, broken when it is slow.

Each pause is instead asked what it was waiting for, and answered from what the recorded events themselves saw:

So a 3.8-second gap becomes

# the recording waited 3.8s here for 'Save As' to open
saveas = gui.wait_for_window("Save As", timeout=11.4)

timeout is in seconds, and it is deliberately not the 3.8s the comment reports: it is three times that wait, floored at ten seconds and capped at two minutes, because replay is a different machine — what the recording took once is not what it will take again. The comment is what happened; the timeout is what the script allows. Every generated file says the same in its header, and the number is a plain literal to edit when a slower machine needs longer — see troubleshooting.md.

Inference runs when a script is generated, not when a recording is made — so --regenerate re-analyzes an old recording under whatever rules exist now, and rendering the same file twice cannot compound.

Checks: what makes it a test

A recording of actions alone is not a test. It passes as long as nothing raises, whatever the application actually did — click Save, and a script that never looks at the result passes just as happily against a build where saving silently fails.

Point at what should have changed and press Ctrl+1:

gui.button("Save").click()
saveas = gui.wait_for_window("Save As", timeout=10)

# Check: 'Status' reads 'Saved'
gui.expect_text(role=Role.LABEL, name="Status", equals="Saved")

What comes out depends on what was under the pointer — a checkbox gives expect_checked, a label with something to say gives expect_text, anything else named gives expect_showing. These are pyguitest Session methods, so a generated script depends on nothing but pyguitest — and it needs 0.15.0 or newer, as the install section explains. They name what was wrong instead of raising a bare AssertionError, and each retries until its timeout so a check cannot race a redraw. Full table in docs/recipes.md.

Install

From PyPI

pip install 'pyguitest-recorder[x11,atspi]'          # Linux, XWayland
pip install pyguitest-recorder 'pyguitest[windows]'  # Windows
pip install 'pyguitest-recorder[macos]'              # macOS

x11 brings python-xlib, which capture needs on Linux. atspi is what lets a click be recorded as a name instead of a coordinate there, through pyguitest's own AT-SPI backend.

On native Windows, capture needs no extra at all — it's pure ctypes — so pyguitest-recorder alone gives you coordinate-only recording; pyguitest [windows] adds named elements, through UI Automation rather than AT-SPI.

On macOS, pyguitest-recorder[macos] is self-contained: it re-exports pyguitest's own macos extra, so the two packages' PyObjC/Quartz dependency can't drift apart.

From a clone

To work on the recorder itself, or track main ahead of a release:

git clone https://github.com/ctrondlp/pyguitest-recorder
cd pyguitest-recorder
pip install -e '.[x11,atspi,dev]'

pyguitest-recorder --doctor

To run without installing anything, put src/ on the import path — every flag works identically:

PYTHONPATH=src python3 -m pyguitest_recorder --doctor

The part pip cannot do for you

dogtail, which element resolution goes through, declares no dependencies of its own: PyGObject and pyatspi have to come from your distribution. Miss them and nothing errors — --doctor reports element resolution off and every click in every recording comes out as a coordinate, which looks like the recorder being bad at its job rather than a missing package.

On Fedora python3-gobject python3-pyatspi at-spi2-core; on Debian and Ubuntu python3-gi python3-pyatspi gir1.2-atspi-2.0. pyguitest's install guide carries the full table, including Arch, openSUSE and FreeBSD.

pyguitest 0.15.0 or newer is required outright — the floor the generated code is verified against. Generated scripts call the expect_ family as pyguitest Session methods, which do not exist before 0.9.0; double-click a named element with Element.double_click, which 0.10.0 added; once motion is set to "natural" or "recorded", move the pointer with Session.move_mouse_naturally, which 0.10.1 added; and 0.11.0 is the first release that imports on Microsoft Windows at all. Element.expand()/ .collapse() and reading .selectable directly, which 0.12.0 added, are what the floor required until 0.14.0 raised it again: that is the release whose macos backend a macOS recording resolves windows and elements through, and whose macquartz vocabulary its key names are translated through, so a recording made on a Mac has nothing older to replay or regenerate against. 0.15.0 is where the floor stands now, and it is the one step in that list nothing here strictly needs: it is the pyguitest this output was last verified against, which is the standard this floor has always used. Nothing here depends on that going unnoticed: validate() checks gui.* calls against the installed Session and what is called on an element against the installed Element, so a script naming a method this pyguitest does not have is reported INVALID when it is generated. --doctor prints the installed pyguitest next to the generator profile, which is what the emitted calls were checked against.

The older floors still matter for what the scripts do. 0.5.0 is where role lookups learned to accept both spellings of a renamed role, and where Window.app_id starts being populated on X11, which is what lets a window whose title drifts still be found; earlier versions lack element_at/extents and double_click as well.

Privacy

Keyboard capture through XRecord sees every application's keystrokes, not only the one you are recording — including your password manager.

• Text typed into an AT-SPI password field is detected and never written into the generated script; it gets os.environ["SECRET_1"] instead. A check recorded against a password field is redacted the same way.
• --sensitive treats all text that way.
• Raw event logs are off unless --record-raw is passed.
• A saved recording is a credential-bearing artifact. Treat it like one — redaction happens when a script is generated, not when events are saved.

Configuration

$XDG_CONFIG_HOME/pyguitest-recorder/config.toml wherever that is set. With it unset the default is per-platform — ~/.config/pyguitest-recorder/config.toml on Linux and the BSDs, %APPDATA%\pyguitest-recorder\config.toml on Windows, since ~/.config is neither conventional nor discoverable there. Either way ~/.pyguitest-recorder.toml is read if the first is absent. Precedence is defaults → file → command line. See config.example.toml.

Status

Early, but the engine is complete and every X11 path has been run live — including capture of a real application, AT-SPI element resolution against a real accessibility bus, and focus-based targeting for typed text. scripts/live-capture-check.py runs the whole pipeline against a private Xvfb on every push, and it has found two bugs that no unit test could have.

Windows recording is newer, but has now captured a real keystroke. A SetWindowsHookExW hook, driven against a purpose-built native window and against real Windows 11 Notepad, recorded and correctly replayed typed text, clicks, menus, tabs, a dropdown, a window drag mid-recording, and a maximize/restore cycle. Its SysListView32 selection carries the one asterisk: the mechanism passed on its own, and the single most complex full run did not reproduce it — left open rather than claimed fixed. scripts/win32-live-capture-check.py is the Windows counterpart of live-capture-check.py. See docs/developers/status.md for what those runs found and fixed, and what is still outstanding.

The per-part verification table, and the known gaps — no UI yet, two recordings of a real desktop application (one on GhostBSD, one of Windows 11 Notepad), and what GTK4 hit-testing costs — are in docs/developers/status.md.

Documentation

• docs/getting-started.md — from nothing to a test you can run
• docs/recipes.md — every flag that matters, by the task it serves
• docs/troubleshooting.md — "why is my script all coordinates?", and the rest
• docs/testable-guis.md — how to build a GUI that can be tested at all; written to be handed to application developers
• docs/developers/architecture.md and docs/developers/status.md — why Wayland has no capture backend, the element-resolution rules, and what has actually been run

License

GPL-2.0-or-later, the same as pyguitest — which this imports at run time and generates source for. See LICENSE.