pyguitest-recorder

Record desktop GUI activity and generate pyguitest scripts from it.

The point is not to replay a macro. It is to turn what you did into test code you would have been willing to write by hand — so a recorded click on a button becomes

gui.button("Save").click()

rather than gui.move_mouse(180, 90); gui.click(), which stops working the moment the window moves, the theme changes, or someone adds a toolbar item.

Quick start

pip install 'pyguitest-recorder[x11,atspi]'          # Linux, XWayland
pip install pyguitest-recorder 'pyguitest[windows]'  # Windows
pip install 'pyguitest-recorder[macos]'              # macOS

pyguitest-recorder --doctor              # can this machine record? why not?
pyguitest-recorder -o login_test.py      # record until Escape, Escape
python3 login_test.py                    # replay it (python on Windows)

⚠️ Keyboard capture sees every application's keystrokes, not only the one you are recording — including your password manager. That is what X11's RECORD extension, Windows' low-level input hooks and macOS's event tap all do, and it is why this tool exists at all. Close what you would not want in a file, and read Privacy before recording anything that touches a login.

Recording needs X11 or XWayland on a Linux desktop, native Windows, or macOS. On Linux that means no Wayland session will do: the recorder reaches XWayland clients and nothing else, and says so rather than producing a file with silent gaps. Why that is permanent.

Three flags carry most of the value:

pyguitest-recorder -o login_test.py --save-session rec.json   # keep both
pyguitest-recorder --regenerate rec.json -o out.py            # re-render, no recording
pyguitest-recorder --record-motion                            # capture hovers too

Recording stops on Escape pressed twice, not only Ctrl-C — a recorder you can only stop from its own terminal is one you cannot stop while driving a full-screen application. Ctrl+1 records a check on whatever the pointer is over. Both are rebindable; see docs/recipes.md.

docs/getting-started.md walks through all of this properly.

What comes out

"""Generated by pyguitest-recorder. Edit freely.

Profile:     pyguitest-0.18
Recorded on: x11 (mutter)
Timeouts:    seconds; the wait the recording observed, rounded up (floor 10s, cap 300s)
"""

import pyguitest
from pyguitest import Capability, Role


def main() -> None:
    """Replay the recorded interaction."""
    with pyguitest.connect() as gui:
        gui.require(
            Capability.ELEMENT_ACTION,
            Capability.ELEMENT_TREE,
            Capability.WINDOW_ACTIVATE,
        )

        editor = gui.wait_for_window("Example", timeout=10)
        gui.activate_window(editor)
        gui.text_field("Name").set_text("Ada")
        gui.button("Save").click()


if __name__ == "__main__":
    main()

Plain pyguitest source, depending on nothing from this package. Three things about it are deliberate.

Elements lead, coordinates follow

Each click is resolved at record time against the accessibility tree, and falls down a ladder only as far as it has to:

    gui.button("Save").click()      a named element — survives redesigns,
                                    themes, resizes and added toolbar items
        ↓  nothing accessible under the pointer?
    window-relative coordinates     survives the window moving
        ↓  no window accounts for the point?
    gui.move_mouse(842, 612)        absolute — breaks when anything moves

--absolute-coordinates forces the bottom rung; --relative-coordinates forces the middle one.

The recorder refuses to name an element it cannot corroborate — if the element's process does not own the window under that point, or its own extents do not contain the point it was looked up at, the click becomes a coordinate instead. A coordinate that works beats a named element that does not. Every such refusal is written into the generated file's docstring, because "why is this script all coordinates?" is the first thing its reader asks. The rules, and the GTK4 measurements behind them, are in docs/developers/architecture.md; the fix when the answer is your own application is docs/testable-guis.md.

Typed text is the exception, and gets named anyway. A run of typing asks the toolkit what has focus rather than what is under the pointer — focus involves no geometry, so it still works where hit-testing has failed. A GTK4 application whose clicks all degrade to coordinates still produces gui.text_field("Name").set_text("Ada"), including for a field reached by Tab or focused by the application itself.

Scripts declare what they need

Every file opens with gui.require(...) naming the capabilities it uses, so a recording made on X11 and replayed somewhere weaker fails on the first line with a typed exception instead of halfway through with a click that went nowhere.

And the file is checked before it is offered. Generation compiles the script, confirms every gui.<method> call exists on the installed pyguitest.Session, and checks each Capability and Role constant against the same — because a recorder that emits a plausible script naming a function the library does not have is worse than no recorder.

Waits, not sleeps

The difference between a recorder and a macro player is what happens to the three seconds you spent waiting for a dialog. A macro player sleeps for three seconds: slow when the machine is fast, broken when it is slow.

Each pause is instead asked what it was waiting for, and answered from what the recorded events themselves saw:

So a 12.4-second gap becomes

# the recording waited 12.4s here for 'Save As' to open
saveas = gui.wait_for_window("Save As", timeout=13)

timeout is in seconds, and it is the wait that comment reports, rounded up to the next whole second — 12.4 becomes 13 — floored at ten seconds and capped at five minutes. Nothing is multiplied on the way out: read the number and you are reading how long the recording itself waited, which is what lets the comment and the call be checked against each other instead of looking like a bug.

The trade is deliberate. Headroom for a slower machine used to be baked into every number, which also hid how long each wait was; now a replay machine much slower than the one that recorded surfaces as a timeout rather than being absorbed. Every generated file says the same in its header, and the number is a plain literal when it needs raising — see troubleshooting.md.

Inference runs when a script is generated, not when a recording is made — so --regenerate re-analyzes an old recording under whatever rules exist now, and rendering the same file twice cannot compound.

Checks: what makes it a test

A recording of actions alone is not a test. It passes as long as nothing raises, whatever the application actually did — click Save, and a script that never looks at the result passes just as happily against a build where saving silently fails.

Point at what should have changed and press Ctrl+1:

gui.button("Save").click()
saveas = gui.wait_for_window("Save As", timeout=10)

# Check: 'Status' reads 'Saved'
gui.expect_text(role=Role.LABEL, name="Status", equals="Saved")

What comes out depends on what was under the pointer — a checkbox gives expect_checked, a label with something to say gives expect_text, anything else named gives expect_showing. These are pyguitest Session methods, so a generated script depends on nothing but pyguitest — and it needs 0.16.2 or newer, as the install section explains. They name what was wrong instead of raising a bare AssertionError, and each retries until its timeout so a check cannot race a redraw. Full table in docs/recipes.md.

Install

From PyPI

pip install 'pyguitest-recorder[x11,atspi]'          # Linux, XWayland
pip install pyguitest-recorder 'pyguitest[windows]'  # Windows
pip install 'pyguitest-recorder[macos]'              # macOS

x11 brings python-xlib, which capture needs on Linux. atspi is what lets a click be recorded as a name instead of a coordinate there, through pyguitest's own AT-SPI backend.

On native Windows, capture needs no extra at all — it's pure ctypes — so pyguitest-recorder alone gives you coordinate-only recording; pyguitest [windows] adds named elements, through UI Automation rather than AT-SPI.

On macOS, pyguitest-recorder[macos] is self-contained: it re-exports pyguitest's own macos extra, so the two packages' PyObjC/Quartz dependency can't drift apart.

From a clone

To work on the recorder itself, or track main ahead of a release:

git clone https://github.com/ctrondlp/pyguitest-recorder
cd pyguitest-recorder
pip install -e '.[x11,atspi,dev]'

pyguitest-recorder --doctor

To run without installing anything, put src/ on the import path — every flag works identically:

PYTHONPATH=src python3 -m pyguitest_recorder --doctor

The part pip cannot do for you

dogtail, which element resolution goes through, declares no dependencies of its own: PyGObject and pyatspi have to come from your distribution. Miss them and nothing errors — --doctor reports element resolution off and every click in every recording comes out as a coordinate, which looks like the recorder being bad at its job rather than a missing package.

On Fedora python3-gobject python3-pyatspi at-spi2-core; on Debian and Ubuntu python3-gi python3-pyatspi gir1.2-atspi-2.0. pyguitest's install guide carries the full table, including Arch, openSUSE and FreeBSD.

pyguitest 0.18.0 or newer is required outright — the floor the generated code is verified against. Generated scripts call the expect_ family as pyguitest Session methods, which do not exist before 0.9.0; double-click a named element with Element.double_click, which 0.10.0 added; once motion is set to "natural" or "recorded", move the pointer with Session.move_mouse_naturally, which 0.10.1 added; and 0.11.0 is the first release that imports on Microsoft Windows at all. Element.expand()/ .collapse() and reading .selectable directly, which 0.12.0 added, are what the floor required until 0.14.0 raised it again: that is the release whose macos backend a macOS recording resolves windows and elements through, and whose macquartz vocabulary its key names are translated through, so a recording made on a Mac has nothing older to replay or regenerate against. 0.15.0 was the step nothing here strictly needed -- the pyguitest this output was last verified against, and no more. 0.15.1 was the first floor step in a while an older install can genuinely get wrong rather than merely lack: a generated script under locators = "element" (the default) scopes an element search with gui.window_element(title), and an older pyguitest there could resolve that call to a shell-owned decoration proxy instead of the real window on a real GNOME/Mutter desktop, found live. 0.16.0 is the release this output is verified against now, and it is not a quiet step either -- five calls in a generated script behave differently there. gui.tap_key(...) on Windows now carries the scan code and extended bit the layout gives the key, where Home, an arrow or Right Ctrl went out as a key no keyboard sends; expect_checked reads a selected radio button as checked, not as None; macOS double_click() counts its clicks, so a Cocoa application sees a double click; gui.drag posts drag events instead of moves; and find_window/wait_for_window name the topmost of two windows sharing a title rather than the one behind. 0.16.1 is a patch step with the same character: on Linux, Element.click() on a GTK switch or toggle button did nothing, and the input backend's pointer landed one pixel short of an absolute position, so a recorded coordinate click missed by one. 0.16.2 is a patch step with the same character again: on macOS Element.text answered from a cache, so a generated expect_text could assert what a field held before the recording's own write reached it, and on Linux windows() listed the override-redirect popups a window manager never manages, so a click could resolve to a drop-down instead of the application's window. validate() checks gui.* calls against the installed Session and what is called on an element against the installed Element, so a script naming a method this pyguitest does not have is reported INVALID when it is generated. --doctor prints the installed pyguitest next to the generator profile, which is what the emitted calls were checked against.

The older floors still matter for what the scripts do. 0.5.0 is where role lookups learned to accept both spellings of a renamed role, and where Window.app_id starts being populated on X11, which is what lets a window whose title drifts still be found; earlier versions lack element_at/extents and double_click as well.

Privacy

Keyboard capture — XRecord on X11 and XWayland, a low-level input hook on Windows, an event tap on macOS — sees every application's keystrokes, not only the one you are recording, including your password manager.

• Text typed into an AT-SPI password field is detected and never written into the generated script; it gets os.environ["SECRET_1"] instead. A check recorded against a password field is redacted the same way.
• --sensitive treats all text that way.
• Raw event logs are off unless --record-raw is passed.
• A saved recording is a credential-bearing artifact. Treat it like one — redaction happens when a script is generated, not when events are saved.

Configuration

$XDG_CONFIG_HOME/pyguitest-recorder/config.toml wherever that is set. With it unset the default is per-platform — ~/.config/pyguitest-recorder/config.toml on Linux and the BSDs, %APPDATA%\pyguitest-recorder\config.toml on Windows, since ~/.config is neither conventional nor discoverable there. Either way ~/.pyguitest-recorder.toml is read if the first is absent. Precedence is defaults → file → command line. See config.example.toml.

Status

Early, but the engine is complete and every X11 path has been run live — including capture of a real application, AT-SPI element resolution against a real accessibility bus, and focus-based targeting for typed text. scripts/live-capture-check.py runs the whole pipeline against a private Xvfb on every push, and it has found two bugs that no unit test could have.

Windows recording is newer, but has now captured a real keystroke. A SetWindowsHookExW hook, driven against a purpose-built native window and against real Windows 11 Notepad, recorded and correctly replayed typed text, clicks, menus, tabs, a dropdown, a window drag mid-recording, and a maximize/restore cycle. Its SysListView32 selection carries the one asterisk: the mechanism passed on its own, and the single most complex full run did not reproduce it — left open rather than claimed fixed. scripts/win32-live-capture-check.py is the Windows counterpart of live-capture-check.py. See docs/developers/status.md for what those runs found and fixed, and what is still outstanding.

macOS is newer still, and has been through the same round trip. A listen-only CGEventTap captured real events from a granted macOS 26.7 Mac, and a record → generate → replay → re-record cycle on that machine compared two recordings event by event: pointer, buttons, scroll, chords and typed text all came back, with windows and elements resolving for both halves of a click. What no live run has shown yet is a person's own keyboard — the tap skips the installing process's own events, so every check so far posted them from a separate process, and no real hardware key has been through it. scripts/ has no macOS counterpart of the two live-check scripts above.

The per-part verification table, and the known gaps — no UI yet, two recordings of a real desktop application (one on GhostBSD, one of Windows 11 Notepad), and what GTK4 hit-testing costs — are in docs/developers/status.md.

Documentation

• docs/getting-started.md — from nothing to a test you can run
• docs/recipes.md — every flag that matters, by the task it serves
• docs/troubleshooting.md — "why is my script all coordinates?", and the rest
• docs/testable-guis.md — how to build a GUI that can be tested at all; written to be handed to application developers
• docs/developers/architecture.md and docs/developers/status.md — why Wayland has no capture backend, the element-resolution rules, and what has actually been run

License

GPL-2.0-or-later, the same as pyguitest — which this imports at run time and generates source for. See LICENSE.