pytest-verifier

A pytest plugin providing soft assertions for test verification. Failed checks don't stop the test unless you ask them to: all checks run to completion, and failures are reported together at test end.

Until 0.5 it was called pytest-verify. See Upgrading from pytest-verify.

Installation

pip install pytest-verifier

This installs the pytest-verifier distribution and the pytest_verifier package. pytest loads the plugin automatically. When plugin autoloading is disabled (PYTEST_DISABLE_PLUGIN_AUTOLOAD), load it with -p pytest_verifier; to turn it off for one run, use -p no:pytest_verifier.

Requires Python 3.9+, pytest 7+ and pluggy 1.2+. Releases up to 0.7.0 were not on PyPI; to install one of them, or the development version, use Git:

pip install "git+https://github.com/guillegil/pytest-verifier.git@v0.7.0"
pip install "git+https://github.com/guillegil/pytest-verifier.git"

Quick Start

Use the verify fixture in any test — no imports needed:

def test_power_supply(verify):
    verify.approx(measured_voltage, 3.3, abs_tol=0.05, name="Vout", units="V")
    verify.greater(throughput, 100, name="Throughput", units="Mbps")
    verify.between(current, 0.1, 0.5, name="Icc", units="A")

If any check fails, the test continues running. When the test body ends, all failures are reported together in a single ChecksFailedError. To stop a test at a check whose failure makes the rest meaningless, see Stopping a test at a failed check.

Failure Output

When one or more checks fail, the test is reported as failed with a summary. Its first line names the first failed check, so -r summaries and junit reports say what failed. Then it lists the failed checks (✗) before the passed ones (✓), each with its index, name, where a failed check was made, and an expected … got … detail:

1 of 3 checks failed: Vout — expected 3.3V ± 0.05V, got 3.8V

  ✗ [1] Vout (tests/test_psu.py:3) — expected 3.3V ± 0.05V, got 3.8V

  ✓ [0] PSU stable — True
  ✓ [2] Throughput — 120Mbps > 100Mbps

A failed check shows the file and line that made it, relative to the rootdir. When a helper made it, the line of the test that called the helper follows: (lib/rails.py:8, called from tests/test_psu.py:22), or called from line 22 when both are in the same file. With --tb=line, pytest points at that line of the test.

How values read in the summary:

• Numbers read naturally, with their units: 3.3V. Strings are quoted, so expected 1, got '1' shows that a reply was never converted. Enum members show as Mode.ACTIVE, and numeric ones add their value: Gain.LOW (10dB).
• When two values still look the same, their types are added: expected 0.1 (float), got 0.1 (Decimal).
• A long value is shortened to about 240 characters. When a failed equal shows the same text for both values, it says where they first differ: first difference at [25]: expected 3.3V, got 3.9V.
• is_true and is_false show the value and how it tests, e.g. '0' (truthy).
• A composite says what went wrong inside it: the first three failing items of an all_satisfy, or the cases and branches it considered when none matched.
• A NaN gets a note, since it never compares equal and fails every ordering: expected 1.0, got nan (NaN never compares equal).
• Each check stays on one line. Line breaks and other control characters in names and values are escaped (\n).
• At most 10 passed checks are listed, followed by ✓ … N more passed checks. Run pytest with -vv to list them all. Every check is still recorded (see Reading Results from Another Plugin).
• On a terminal that cannot show ✗ and ✓, such as a Windows CI log, they print as x and ok, and any other character the terminal cannot show is escaped (\u2014). Only the terminal output changes: reports such as junitxml keep the summary as it is.

When failures are raised

ChecksFailedError is an AssertionError, so pytest treats a soft failure like a failed assert, and xfail(raises=AssertionError) catches it.

• Checks made in fixtures' setup and in the test body are raised after the test body.
• Checks made while fixtures are torn down are raised after teardown, as a teardown error. Their indices follow those of the test body.
• If the test already failed with another exception, that exception is reported and the failed checks are added to its report under Soft assertion failures. This works in unittest TestCases too.
• A skip after a failed check does not hide the failure: pytest.skip(), unittest.SkipTest and TestCase.skipTest() alike.
• --pdb opens the debugger when the failure is raised. For soft checks that is after the test body has finished, so the test's local variables are gone. To inspect them, check the returned result, e.g. if not check["passed"]: breakpoint(), or make the check required (verify.require, or --verify-fail-fast for every check): then --pdb opens in the test, at the line of the failed check.
• Rerun filters match exceptions by name, so use --only-rerun ChecksFailedError with pytest-rerunfailures.

Stopping a test at a failed check

Some failures make the rest of a test meaningless, such as a link that could not be opened. Make that check required: verify.require has the same methods as verify, and a check made with it that fails stops the test at once with ChecksFailedError, listing every check made so far.

def test_link(verify):
    verify.equal(read_firmware(), "1.2.0", name="Firmware")
    link = open_link()
    verify.require.is_not_none(link, name="Link")   # stops here when there is no link
    verify.equal(link.status(), "ready", name="Status")

The first line of the summary then names the check that stopped the test, and --tb=line points at its line:

2 of 2 checks failed, stopped at [1]: Link — expected not None, got None (+1 more)

  ✗ [0] Firmware (tests/test_link.py:2) — expected '1.2.0', got '1.1.0'
  ✗ [1] Link (tests/test_link.py:4) — expected not None, got None

verify.require(check) does the same for a check made earlier or built with checks, for example verify.require(verify.equal(reply, "OK", name="Reply")). The check stays recorded, so a test that catches the error still fails. A required check stops the test from inside a lazy child or an all_satisfy factory too. In a fixture, a required check that fails is an error in the test's setup or teardown, like a failed assert there. To pass verify.require to a helper, annotate the parameter as pytest_verifier.Require.

To stop every test at its first failed check, for example while bringing up new hardware, run pytest with --verify-fail-fast, or turn it on in the configuration:

[tool.pytest.ini_options]
verify_fail_fast = true

Fail-fast leaves checks made while fixtures are torn down soft, so a fixture's cleanup after a failed check still runs, and so are the checks of a unittest TestCase's tearDown, asyncTearDown and cleanups; use verify.require to stop there. In a fixture's setup, a failed check stops like an assert: if that happens before yield, the fixture's teardown does not run either, so put cleanup that must run in try/finally or request.addfinalizer.

Every check is judged when it is made. With fail-fast, a check passed directly as a conditional case or default, or a guard branch or default, is judged before the composite chooses, so a failed one stops the test even when it would not be selected. That includes default=verify.fail(...), which always fails, and the checks an all_satisfy factory makes. Pass cases, branches and defaults as functions (see Lazy children) so that only the selected one is judged.

Checks that cannot be evaluated

A comparison that raises never stops the test by itself: the check fails and the error is shown (a required or fail-fast check then stops the test like any failed check), for example comparing None with a number:

  ✗ [1] Reading (tests/test_power.py:18) — expected > 100, got None (TypeError: '>' not supported between instances of 'NoneType' and 'int')

Text is never compared as a number. "100" < "20" is true for Python, which compares strings letter by letter, so the ordering checks (greater, less, their _equal forms and between) fail when the value and a limit are both a str, bytes or bytearray. Convert instrument replies and values read from files first, e.g. float(reply):

  ✗ [2] Ripple (tests/test_power.py:19) — expected < '20', got '100' (TypeError: str values are compared as text, not as numbers; convert readings with float() first)

Text against a number fails with Python's own error, plus the same advice. Values that compare with text on their own terms, such as a semver.Version against "1.9.0", work as usual.

The same happens when a comparison returns something whose truth value is ambiguous, such as a numpy array. Reduce it first: verify.is_true((a == b).all(), name="Arrays equal").

Mistakes in how a check is called raise right away, like any Python error: a missing or non-string name, a negative tolerance, between with low above high, or a malformed guard branch.

Check Functions

Equality & Approximation

With units="%", the tolerance says whether it is absolute or relative: 50% ± 1% (abs).

Ordering & Range

These compare numbers, or other values that order themselves such as version tuples and dates. Text compared with text fails (see Checks that cannot be evaluated).

Boolean & Identity

String & Container

Type, Collection & Conditional

The fixture also has verify.record(check), which records a check built elsewhere (see Recording checks built by helpers), and verify.require, whose checks stop the test when they fail (see Stopping a test at a failed check).

Usage Examples

Most checks read like their table entry — verify.equal(status, 200, name="Status"). The ones below take a little more setup.

conditional — pick one branch by a switch value

Only the case whose key matches switch_value counts. Use default for the no-match case; without one, no match is a failure. A key matches when it equals the switch value. Enum members match by their value, and an int matches its decimal string, so cases={0: ..., 1: ...} and cases={"0": ..., "1": ...} behave the same. Keys that would match the same values, such as 1 and "1", raise ValueError. When nothing matched and there is no default, the summary lists the keys it tried: [mode=7 → no case matched: 0, 1, 2].

def test_output_by_mode(verify):
    verify.conditional(
        mode,                       # e.g. 0, 1, or 2
        name="Output voltage",
        cases={
            0: verify.approx(output, 0.0, abs_tol=0.01, name="Standby", units="V"),
            1: verify.approx(output, 3.3, abs_tol=0.1, name="Active", units="V"),
            2: verify.approx(output, 5.0, abs_tol=0.1, name="Boost", units="V"),
        },
        default=lambda: verify.fail(f"Unknown mode: {mode}"),
    )

The default is a function, so it fails only when it is selected (see Lazy children). With --verify-fail-fast, make the cases functions too.

guard — if / elif / else with arbitrary conditions

When the expected check depends on a chain of conditions (not a single switch value), list them as ordered (condition, label, check) branches. The first branch whose condition is true is evaluated; if none match, default is used. The label identifies the chosen branch in the failure summary.

def test_sensor_output(verify):
    verify.guard(
        branches=[
            (shutter_closed,     "shutter closed", verify.equal(reading, 0, name="Dark")),
            (level < floor - 5,  "below floor",    verify.equal(reading, 0, name="Dark")),
            (not sensor_enabled, "disabled",       verify.equal(reading, 0, name="Dark")),
        ],
        default=verify.approx(reading, expected_dn, abs_tol=2, name="Lit", units="DN"),
        name="Sensor output",
    )

Conditions can be any truthy or falsy value, and a condition that is a function is called (see Lazy children). A failed guard reports the branch it took, e.g. ✗ [0] Sensor output (tests/test_sensor.py:2) [→ below floor] — expected 0, got 7, or the labels it tried when none matched: [→ no branch matched: shutter closed, below floor]. When a condition raises, the guard fails with [→ no branch chosen] and the error. Passing a check as a condition raises TypeError, because a check is always truthy. Use its result instead, e.g. check["passed"].

all_satisfy — apply one check to every item

The factory is called once per item to build a child check; the parent passes only if all children pass. items can be any iterable. If the factory raises or returns something that is not a check (for example, a forgotten return), the check fails with that error.

def test_all_channels(verify):
    verify.all_satisfy(
        channel_voltages,                                  # e.g. [3.31, 3.29, 3.30]
        lambda v: verify.between(v, 3.2, 3.4, name="Channel", units="V"),
        name="All channels within spec",
    )

A failure names the first three failing items by their index:

  ✗ [0] All channels within spec (tests/test_channels.py:2) — expected all 4 to pass, got 2 failed: [1] expected [3.2V, 3.4V], got 3.55V; [3] expected [3.2V, 3.4V], got 3.1V

How child checks are counted

With the fixture, every check you pass to a composite (in cases, branches, default, or built by the all_satisfy factory) belongs to the composite. It is not reported on its own, and only the selected ones count toward the composite's verdict. This holds whether you build the checks inline or earlier, for example in a cases dict:

cases = {
    0: verify.approx(output, 0.0, abs_tol=0.01, name="Standby", units="V"),
    1: verify.approx(output, 3.3, abs_tol=0.1, name="Active", units="V"),
}
verify.conditional(mode, cases=cases, name="Output voltage")  # only the selected case counts

A child built this way is evaluated when it is built, because it is an argument of the call, so it should not depend on values that only its own branch can use. A child that raises is just a failed child. When that matters, build the children lazily (see below).

To also keep a check on its own, pass a copy: verify.guard([(cond, "label", dict(check))], ...). A check that stopped the test (see Stopping a test at a failed check) always stays on its own as well, so the summary can name it and a test that catches the error still fails; a composite made afterwards that selects it counts it once more. A composite built with checks (see Building checks without the fixture) is never recorded, so fixture checks passed to it stay separate checks.

Lazy children — build only the selected branch

A conditional case or default, and a guard check or default, can be a function with no arguments that returns the check, such as a lambda. Only the selected one is called, so the other branches never touch values they cannot use. A guard condition can be a function too. Conditions are called in order until one is true, and the ones after it are not called. A condition function that returns a check instead of a truth value makes the guard fail with an error, because a check is always truthy.

def test_sensor_output(verify):
    verify.guard(
        branches=[
            (lambda: sensor.shutter_closed(), "shutter closed",
             lambda: verify.equal(sensor.read(), 0, name="Dark")),
            (lambda: sensor.enabled(), "enabled",
             lambda: verify.approx(sensor.read(), 512, abs_tol=2, name="Lit", units="DN")),
        ],
        default=lambda: verify.fail("sensor disabled"),
        name="Sensor output",
    )

In the recorded check, a lazy child that was not called is None, and so is a condition that was not called. If the selected function raises, or returns something that is not a check (for example, a forgotten return), the composite fails and its error says why. Any other check the function records while it runs stays a check of its own.

is_instance — type check

Takes what isinstance takes: a class, a tuple of classes, or a union.

verify.is_instance(response, dict, name="Response is a dict")
verify.is_instance(reading, (int, float), name="Reading is a number")

fail — force a failure

Useful as the default branch of a conditional, or to mark an unreachable path. name defaults to the message. As a default, pass it as a function, default=lambda: verify.fail("..."): a check made directly is recorded as failed at once, and with --verify-fail-fast it stops the test even when a case matches.

verify.fail(f"Unexpected state: {state}")

Building checks without the fixture

checks has the same methods as the fixture, but it only builds checks: each call returns an unevaluated descriptor, and nothing is recorded. Use it in helper functions, or to evaluate checks yourself:

from pytest_verifier import checks

descriptor = checks.approx(3.28, 3.3, abs_tol=0.05, name="Vout")
result = checks.evaluate(descriptor)       # True / False
details = checks.evaluate_detailed(descriptor)  # [{passed, details, seq, t}]

Pass several checks as separate arguments: checks.evaluate(*descriptors). Each result of evaluate_detailed also has an error key when its check could not be evaluated.

A check built with checks cannot fail a test by itself. If the body of a test builds one and it is not recorded, evaluated or passed to a composite by the end of the test's teardown, pytest shows an UnusedCheckWarning that points at the line that built it. A fixture can therefore collect checks from the test and record them when it is torn down. Checks built in fixtures, at import time or in a unittest TestCase are not tracked, so a fixture can prepare checks for later tests. Where building checks without using them is intended, filter the warning, for the whole project or for one module:

[tool.pytest.ini_options]
filterwarnings = ["ignore::pytest_verifier.UnusedCheckWarning"]
# or only in tests/test_limits.py:
# filterwarnings = ["ignore::pytest_verifier.UnusedCheckWarning:tests.test_limits"]

With -W error::pytest_verifier.UnusedCheckWarning, the warning fails the test's teardown.

A descriptor sent through JSON loses Python types: tuples become lists and dict keys become strings. equal((1, 2), [1, 2]) fails, but the same descriptor passes after a JSON round-trip. Results recorded by the fixture carry their passed verdict, and evaluate() keeps it.

Recording checks built by helpers

A helper that builds checks with checks does not need the fixture. In the test, pass what it returns to the fixture's verify.record(). The check is then judged and reported like any other, and a composite absorbs the fixture checks passed to it.

# helpers.py
from pytest_verifier import checks

def rail_ok(voltage):
    return checks.between(voltage, 3.2, 3.4, name="3V3 rail", units="V")

# test_power.py
def test_rails(verify):
    verify.record(rail_ok(measure("3V3")))

Calling checks.record() raises RuntimeError, because only the fixture records checks.

Reading Results from Another Plugin

Reporters and other plugins read a test's checks with get_check_results(item):

from pytest_verifier import get_check_results

def pytest_runtest_makereport(item, call):
    for check in get_check_results(item):
        print(check["name"], check["passed"], check["detail"])

It returns a copy of the checks the test recorded, in order. Each one is a plain dict with passed, detail, phase ("setup", "call" or "teardown", the test phase that made it) and location ("tests/test_psu.py:3", where it was made, relative to the rootdir). When a helper made the check, called_from holds the line of the test function that led to it. Each check holds JSON-safe copies of the checked values taken when the check was made, so json.dumps works on it. A check nested in all_satisfy, conditional or guard is inside its parent. After a rerun, only the last attempt's checks are returned. pytest-reporter uses this to render verification cards.

A plugin that should not import pytest-verifier can implement the pytest_verify_results hook instead. It is called when a phase ends with checks to judge: when the test body ends, for the checks made in setup and in the body, and when teardown ends, for the checks made in teardown. If setup fails or skips, it is called with when="setup" for the checks made so far. Mark it optional, so it also loads where pytest-verifier is not installed:

import pytest

@pytest.hookimpl(optionalhook=True)
def pytest_verify_results(item, when, checks, passed):
    print(item.nodeid, when, passed, [check["name"] for check in checks])

The same checks are on that phase's test report as report.verify_checks. They are JSON-safe, so they also reach the main process under pytest-xdist.

Upgrading from 0.7

• pytest-verifier is on PyPI: pip install pytest-verifier.
• The first line of a failure now names the first failed check: 2 of 5 checks failed: Vout — expected 3.3V ± 0.05V, got 3.8V (+1 more). A tool that matched the whole line N of M checks failed should match its start instead.
• Failed checks show where they were made, and recorded checks have the new keys location and called_from.
• With --tb=line, the line shown is the test's line that made the check the first line names instead of the def line.

Upgrading from 0.6

Most of 0.7 changes how failures read. A few changes can affect existing tests:

• An ordering check between two texts, such as verify.greater("100", "20", name=...), now fails with an error instead of comparing letter by letter. Convert readings with float().
• Type checkers see narrower types: length() needs a sized value and contains() a container, both reject an Optional until it is narrowed, and the body of an all_satisfy lambda is checked against the type of the items.
• The summary text has new formats (quoted strings, type hints, at most 10 passed checks unless -vv). Code that needs the results should read them with get_check_results() or report.verify_checks rather than parse the text.

Every change is listed in the CHANGELOG.

Upgrading from pytest-verify

Version 0.6.0 renamed the project, because another plugin on PyPI already uses the name pytest-verify and its pytest_verify package. Tests that only use the verify fixture need no change. The hook pytest_verify_results and report.verify_checks keep their names too.

First uninstall the old distribution, then install the new one:

pip uninstall pytest-verify
pip install pytest-verifier

If pytest-verify is still installed, for example after pip install -U from the same git URL, pytest stops with a message that says so, because the two cannot be loaded together. Then update the imports and options that use the old name:

pytest_verifier.verify still works as an alias of checks and shows a DeprecationWarning. It will be removed in a future release.

Development

The project uses uv. Clone and run the test suite:

git clone https://github.com/guillegil/pytest-verifier.git
cd pytest-verifier
uv run pytest

Type-check with uv run --with mypy mypy (strict mode, configured in pyproject.toml).

CI runs on every push to main and every pull request targeting main (see .github/workflows/ci.yml). It runs the test suite on Python 3.9–3.13 and on the oldest supported pytest (7.0) and pluggy (1.2). It also runs mypy --strict and the tests of the built sdist.

To release, bump version in pyproject.toml, move the [Unreleased] CHANGELOG entries under a dated heading for the new version, merge, then push a vX.Y.Z tag or run the Release workflow (.github/workflows/release.yml). It builds the sdist and wheel once, runs the tests against the wheel with the oldest and the newest pytest, tags the commit, uploads to PyPI with Trusted Publishing, and publishes a GitHub release with the CHANGELOG notes. It only releases commits that are on main. Run it with the testpypi target, from any branch, to try an upload on TestPyPI first. If a run fails half way, use Re-run failed jobs, which reuses the files it built: an index never accepts other files for a version it already has.

Publishing needs a one-time setup: on pypi.org and test.pypi.org, add this repository's release.yml as a trusted publisher with the environment pypi (or testpypi), and in the repository's Settings > Environments, limit pypi to the main branch and v* tags.

Known Issues and Roadmap

Version 0.4.0 fixed every bug found by the review of 0.3.1. The report is in bugs-0.3.1.md, and tests/test_regressions_*.py keeps a regression test for each bug.

Planned improvements and feature ideas are collected in improvements-and-ideas.md. CHECKLIST.md tracks every item and the release that handles it. See CHANGELOG.md for what each release changed.

License

MIT