pyguitest-recorder

Record desktop GUI activity and generate pyguitest scripts from it.

The point is not to replay a macro. It is to turn what you did into test code you would have been willing to write by hand — so a recorded click on a button becomes

gui.button("Save").click()

rather than gui.move_mouse(180, 90); gui.click(), which stops working the moment the window moves, the theme changes, or someone adds a toolbar item.

Quick start

pip install 'pyguitest-recorder[x11,atspi]'

pyguitest-recorder --doctor              # can this machine record? why not?
pyguitest-recorder -o login_test.py      # record until Escape, Escape
python3 login_test.py                    # replay it

⚠️ Keyboard capture sees every application's keystrokes, not only the one you are recording — including your password manager. That is what X11's RECORD extension does, and it is why this tool exists at all. Close what you would not want in a file, and read Privacy before recording anything that touches a login.

Not on PyPI yet — nothing has been released, so until then use the clone below. Recording needs X11 or XWayland; under a Wayland session it reaches XWayland clients and nothing else, and says so rather than producing a file with silent gaps. Why that is permanent.

Three flags carry most of the value:

pyguitest-recorder -o login_test.py --save-session rec.json   # keep both
pyguitest-recorder --regenerate rec.json -o out.py            # re-render, no recording
pyguitest-recorder --record-motion                            # capture hovers too

Recording stops on Escape pressed twice, not only Ctrl-C — a recorder you can only stop from its own terminal is one you cannot stop while driving a full-screen application. Ctrl+F1 records a check on whatever the pointer is over. Both are rebindable; see docs/recipes.md.

docs/getting-started.md walks through all of this properly.

What comes out

"""Generated by pyguitest-recorder. Edit freely.

Profile:     pyguitest-0.5
Recorded on: x11 (mutter)
"""

import pyguitest
from pyguitest import Capability, Role


def main() -> None:
    """Replay the recorded interaction."""
    with pyguitest.connect() as gui:
        gui.require(
            Capability.ELEMENT_ACTION,
            Capability.ELEMENT_TREE,
            Capability.WINDOW_ACTIVATE,
        )

        editor = gui.wait_for_window("Example", timeout=10)
        gui.activate_window(editor)
        gui.text_field("Name").set_text("Ada")
        gui.button("Save").click()


if __name__ == "__main__":
    main()

Plain pyguitest source, depending on nothing from this package. Three things about it are deliberate.

Elements lead, coordinates follow

Each click is resolved at record time against the accessibility tree, and falls down a ladder only as far as it has to:

    gui.button("Save").click()      a named element — survives redesigns,
                                    themes, resizes and added toolbar items
        ↓  nothing accessible under the pointer?
    window-relative coordinates     survives the window moving
        ↓  no window accounts for the point?
    gui.move_mouse(842, 612)        absolute — breaks when anything moves

--absolute-coordinates forces the bottom rung; --relative-coordinates forces the middle one.

The recorder refuses to name an element it cannot corroborate — if the element's process does not own the window under that point, or its own extents do not contain the point it was looked up at, the click becomes a coordinate instead. A coordinate that works beats a named element that does not. Every such refusal is written into the generated file's docstring, because "why is this script all coordinates?" is the first thing its reader asks. The rules, and the GTK4 measurements behind them, are in docs/developers/architecture.md; the fix when the answer is your own application is docs/testable-guis.md.

Typed text is the exception, and gets named anyway. A run of typing asks the toolkit what has focus rather than what is under the pointer — focus involves no geometry, so it still works where hit-testing has failed. A GTK4 application whose clicks all degrade to coordinates still produces gui.text_field("Name").set_text("Ada"), including for a field reached by Tab or focused by the application itself.

Scripts declare what they need

Every file opens with gui.require(...) naming the capabilities it uses, so a recording made on X11 and replayed somewhere weaker fails on the first line with a typed exception instead of halfway through with a click that went nowhere.

And the file is checked before it is offered. Generation compiles the script, confirms every gui.<method> call exists on the installed pyguitest.Session, and checks each Capability and Role constant against the same — because a recorder that emits a plausible script naming a function the library does not have is worse than no recorder.

Waits, not sleeps

The difference between a recorder and a macro player is what happens to the three seconds you spent waiting for a dialog. A macro player sleeps for three seconds: slow when the machine is fast, broken when it is slow.

Each pause is instead asked what it was waiting for, and answered from what the recorded events themselves saw:

So a four-second gap becomes

# the recording waited 3.8s here for 'Save As' to open
saveas = gui.wait_for_window("Save As", timeout=11.4)

with the timeout scaled to what was actually observed rather than guessed.

Inference runs when a script is generated, not when a recording is made — so --regenerate re-analyzes an old recording under whatever rules exist now, and rendering the same file twice cannot compound.

Checks: what makes it a test

A recording of actions alone is not a test. It passes as long as nothing raises, whatever the application actually did — click Save, and a script that never looks at the result passes just as happily against a build where saving silently fails.

Point at what should have changed and press Ctrl+F1:

gui.button("Save").click()
saveas = gui.wait_for_window("Save As", timeout=10)

# Check: 'Status' reads 'Saved'
expect_text(gui, role=Role.LABEL, name="Status", equals="Saved")

What comes out depends on what was under the pointer — a checkbox gives expect_checked, a label with something to say gives expect_text, anything else named gives expect_showing. The expect_ helpers are written into the generated file rather than imported, they name what was wrong instead of raising a bare AssertionError, and each retries until its timeout so a check cannot race a redraw. Full table in docs/recipes.md.

Install

From PyPI

pip install 'pyguitest-recorder[x11,atspi]'

x11 brings python-xlib, which capture needs. atspi is what lets a click be recorded as a name instead of a coordinate.

Not on PyPI yet — nothing has been released, so until then use the clone below.

From a clone

git clone https://github.com/ctrondlp/pyguitest-recorder
cd pyguitest-recorder
pip install -e '.[x11,atspi,dev]'

pyguitest-recorder --doctor

To run without installing anything, put src/ on the import path — every flag works identically:

PYTHONPATH=src python3 -m pyguitest_recorder --doctor

The part pip cannot do for you

dogtail, which element resolution goes through, declares no dependencies of its own: PyGObject and pyatspi have to come from your distribution. Miss them and nothing errors — --doctor reports element resolution off and every click in every recording comes out as a coordinate, which looks like the recorder being bad at its job rather than a missing package.

On Fedora python3-gobject python3-pyatspi at-spi2-core; on Debian and Ubuntu python3-gi python3-pyatspi gir1.2-atspi-2.0. pyguitest's install guide carries the full table, including Arch, openSUSE and FreeBSD.

pyguitest 0.5.0 or newer is required outright. Generated scripts call gui.button(...), which finds nothing on a current at-spi2 before 0.5.0 — that release is where role lookups learned to accept both spellings of a renamed role. 0.5.0 is also where Window.app_id starts being populated on X11, which is what lets a window whose title drifts still be found. Earlier versions lack element_at/extents and double_click as well.

Privacy

Keyboard capture through XRecord sees every application's keystrokes, not only the one you are recording — including your password manager.

• Text typed into an AT-SPI password field is detected and never written into the generated script; it gets os.environ["SECRET_1"] instead. A check recorded against a password field is redacted the same way.
• --sensitive treats all text that way.
• Raw event logs are off unless --record-raw is passed.
• A saved recording is a credential-bearing artifact. Treat it like one — redaction happens when a script is generated, not when events are saved.

Configuration

$XDG_CONFIG_HOME/pyguitest-recorder/config.toml, falling back to ~/.pyguitest-recorder.toml. Precedence is defaults → file → command line. See config.example.toml.

Status

Early, but the engine is complete and every path has now been run — including live capture of a real application, AT-SPI element resolution against a real accessibility bus, and focus-based targeting for typed text. scripts/live-capture-check.py runs the whole pipeline against a private Xvfb on every push, and it has found two bugs that no unit test could have.

The per-part verification table, and the known gaps — no UI yet, only one recording of a real desktop application, and what GTK4 hit-testing costs — are in docs/developers/status.md.

Documentation

• docs/getting-started.md — from nothing to a test you can run
• docs/recipes.md — every flag that matters, by the task it serves
• docs/troubleshooting.md — "why is my script all coordinates?", and the rest
• docs/testable-guis.md — how to build a GUI that can be tested at all; written to be handed to application developers
• docs/developers/ — why recording is X11 only, the element-resolution rules, and what has actually been run

License

GPL-2.0-or-later, the same as pyguitest — which this imports at run time and generates source for. See LICENSE.