pytest-qamule

A pytest plugin for QAMule automation testing, integrating device control, live pause debugging/checkpoint, and AI/human-friendly test reports.

What it does

• Device fixtures: expose uiautomator2 devices as pytest fixtures such as d, phone, or tablet.
• Live pause: pause a failing test, inspect the app state, then resume or abort from the CLI.
• Checkpoints: add explicit human/AI review points inside tests with the checkpoint fixture.
• Live reports: write an event-backed report while pytest runs, then inspect it from a browser or JSON CLI.

Install

uv add pytest-qamule

The package exposes a pytest plugin through the qamule pytest11 entry point and a qamule CLI script.

Device fixtures

pytest-qamule exposes uiautomator2 devices as pytest fixtures. With no options, a default d fixture connects through u2.connect():

pytest

def test_home_screen(d):
    d.app_start("com.example.app")

Register named devices with repeated --device NAME:SERIAL options:

pytest --device phone:emulator-5554 --device tablet:0123456789

def test_home_screen(phone):
    phone.app_start("com.example.app")

Pause Sessions

QAMule creates a session directory directly under the current working directory. Pause protocol files are written under <session-id>/pauses/<pause-id>/. Pytest prints the session id at startup:

QAMule Session ID: s-20260624-153000-a1b2c3
QAMule Session Dir: /path/to/project/s-20260624-153000-a1b2c3

The session is marked finished during pytest unconfigure. Generated s-*/ session directories are ignored by git.

Failure Pause

Enable live pause on test failures with --qamule-pause-on-failure, or use the shorter --pause-on-failure alias for local debugging.

pytest --qamule-pause-on-failure -s
pytest --pause-on-failure -s

Failure pause triggers only when the pytest call phase fails and the report is not an expected xfail. It pauses after the failing call report is produced, then resumes teardown and the rest of the run after a resume command.

Attach preset images before the failure with the auto-loaded qamule_pause fixture:

def test_failure(qamule_pause):
    qamule_pause.add_image("before-failure.png")
    qamule_pause.add_images(["detail-1.png", "detail-2.png"])
    assert False, "boom"

After resume or abort, failure pause data is attached to the pytest call report for future reporting integrations:

def pytest_runtest_logreport(report):
    if report.when == "call" and hasattr(report, "qamule_pause"):
        message = report.qamule_pause["message"]
        images = report.qamule_pause["images"]

The same payload is also added to report.user_properties with the key qamule_pause.

Checkpoints

The checkpoint fixture is auto-loaded by the pytest plugin and can be used directly in tests. It pauses immediately when called, without requiring --pause-on-failure.

def test_checkpoint(checkpoint):
    response = checkpoint("verify app state", images=["checkpoint.png"])
    assert response.result is True
    assert response.message == "checkpoint accepted"
    assert response.images[0].base64

checkpoint(...) returns a CheckpointResult with:

response.result
response.message
response.images

Resume a checkpoint with a result:

qamule pause resume p-000001 --session-id <session-id> --result true --message "checkpoint accepted"

For non-interactive runs, mock checkpoint resume results at pytest startup:

pytest --qamule-checkpoint-mock-result true -s
pytest --qamule-checkpoint-mock-result false -s
pytest --qamule-checkpoint-mock-result none -s
pytest --qamule-checkpoint-mock-result true --qamule-checkpoint-mock-message "approved by reviewer" -s

Mocked checkpoints still write pause and result protocol files, then immediately resume. By default, the checkpoint message is the selected mock value: true, false, or none. Use --qamule-checkpoint-mock-message to attach a more descriptive review message.

--result must be true, false, or none. Checkpoint pauses do not support abort. Checkpoint timeouts automatically resume with response.result is None and response.message == "pause timed out"; they do not automatically fail the test.

CLI

qamule pause commands emit JSON and require --session-id:

qamule pause status --session-id <session-id>
qamule pause ls --session-id <session-id>
qamule pause show <pause-id> --session-id <session-id>
qamule pause resume <pause-id> --session-id <session-id> --message "checked" --image screenshot.png
qamule pause abort <pause-id> --session-id <session-id> --message "stop run" --image screenshot.png
qamule pause watch --session-id <session-id> --timeout 30 --interval 0.5

Command behavior:

• status: prints whether the session has active pauses; exits 0 when active pauses exist and 2 otherwise.
• ls: lists all pauses for the session.
• show: prints one pause by id; exits 2 if the pause is missing.
• resume: writes a resume command for an active pause.
• abort: writes an abort command for an active failure pause; checkpoint abort exits 5 and does not write a command.
• watch: waits for an active pause, a finished session, or timeout; timeout exits 2.

resume continues a pause. abort stops a failure pause by raising KeyboardInterrupt in pytest.

Images

The --image option accepts a file path and can be repeated. File contents are encoded as base64, and the file name is stored as the image alias. Duplicate image content is removed while preserving first-seen order. The same alias can appear more than once when the underlying base64 content is different.

qamule commands keep terminal JSON concise: attached images are reported by alias or count, and base64 fields are omitted from command output. Full base64 image data is written only to the pause protocol files for the report UI and protocol readers.

Images can come from code-side presets and CLI commands. They are merged in this order:

1. Code-side preset images from qamule_pause or checkpoint(..., images=[...])
2. CLI images from qamule pause resume ... --image ... or qamule pause abort ... --image ...

Each image has this shape in protocol files, checkpoint responses, and failure report attachments:

{
    "alias": "image",
    "base64": "...",
}

Example failure workflow:

pytest test_a.py --pause-on-failure -s
qamule pause watch --session-id <session-id>
qamule pause show p-000001 --session-id <session-id>
qamule pause resume p-000001 --session-id <session-id> --message "checked failure state" --image screenshot.png

Example checkpoint workflow:

pytest test_a.py -s
qamule pause watch --session-id <session-id>
qamule pause resume p-000001 --session-id <session-id> --result true --message "checkpoint accepted" --image final.png

Live Reports

QAMule writes the live report store automatically whenever pytest runs:

pytest
pytest --pause-on-failure -s

Report files are written directly in the QAMule session directory while pytest is still running:

<session-id>/
events.jsonl
state.json

events.jsonl is the append-only event stream, and state.json records whether the run is running, finished, aborted, interrupted, or lost. QAMule does not store a duplicated report snapshot; report views are built dynamically from <session-id>/events.jsonl plus pause protocol files under <session-id>/pauses/. Evidence images are stored once in pause protocol JSON as alias and base64, and report views reuse that data instead of creating image asset files or copying base64 into report events. If the pytest process disappears while state.json still says running, report readers display the session as lost instead of treating the report as invalid.

Serve the human-friendly live report UI with:

qamule report serve --session-id <session-id>
qamule report serve --session-id <session-id> --host 127.0.0.1 --port 8765
qamule report serve --session-id <session-id> -q

The report server opens the browser UI by default and exposes JSON endpoints backed by the same dynamically built report view. Pass -q or --quite to print the URL without opening a browser. The browser uses server-sent events to render new results as they arrive. The UI logo is served from packaged report assets, so it does not require a separate logo.jpg in the current working directory.

All qamule report commands read events.jsonl and state.json from <session-id>/ under the current working directory. To inspect a copied report, run the command from the directory that contains the session directory.

AI agents and scripts can query focused slices of the report without reading the full file:

qamule report status --session-id <session-id>
qamule report list --session-id <session-id>
qamule report list --session-id <session-id> --outcome failed
qamule report list --session-id <session-id> --with evidence
qamule report show tests/test_a.py::test_example --session-id <session-id>
qamule report failures --session-id <session-id>
qamule report checkpoints --session-id <session-id>
qamule report rebuild --session-id <session-id>

qamule report list can filter by --outcome passed|failed|skipped|running|unknown and by --with checkpoint|failure-evidence|evidence.

The default report view focuses on test outcomes, checkpoint decisions, failure evidence, messages, results, and images. Detailed event history remains available through <session-id>/events.jsonl, while pause details and images remain in <session-id>/pauses/.

When a pause has both code-side preset images and CLI-attached images, the report keeps the legacy merged images list for compatibility and also exposes preset_images and attached_images so the browser can display them in separate groups.

To avoid duplicate image storage, pause.json owns code-side preset image base64 and command.json owns CLI-attached image base64. result.json records only the pause decision fields and does not store image base64.

Development

Pause and report CLI commands share the same public JSON output helper. When changing CLI output, keep terminal JSON free of base64 fields and cover the shared helper plus the affected feature tests.

uv sync
uv run pytest
uv run pytest tests/unit/test_output.py
uv run pytest tests/unit/features/device tests/integration/features/device
uv run pytest tests/unit/features/pause tests/integration/features/pause
uv run pytest tests/unit/features/report tests/integration/features/report