pytest-xspec

Executable specifications with pytest.

Link specifications to your existing pytest tests, organize your test suite your way, and see which specifications your test suite verifies. Detect missing tests and enforce verification in CI.

Usage

Install pytest-xspec into your pytest environment. The plugin is discovered automatically. Python 3.10+ and pytest 8+ are supported.

pip install pytest-xspec

Declare specifications

Declare specification IDs under requirements in pyproject.toml.

[tool.xspec]
requirements = ["REQ-001", "REQ-002"]

Alternatively, use one ID per line in pytest.ini.

[pytest]
xspec =
    REQ-001
    REQ-002

Declare specifications in one of these files. Using both is an error, and other files do not supply specifications. IDs are opaque strings. Duplicates and marked IDs that have not been declared are rejected. No inventory, or an empty inventory, means zero specifications.

Mark tests

Associate tests with specifications, using one marker per specification.

import pytest


@pytest.mark.xspec("REQ-001")
@pytest.mark.xspec("REQ-002")
def test_shared_behavior():
    assert 1 + 1 == 2

Run pytest

Plain pytest checks specification configuration and markers without printing a specification report. Add --xspec-report to see the report, or --xspec-strict to also fail the run when a specification is unverified.

pytest --xspec-report             # Print the verification report
pytest --xspec-json=report.json   # Export statuses and individual test outcomes
pytest --xspec-strict             # Report and fail on unverified specifications
pytest -m xspec --xspec-strict    # Run only marked specification tests

The report is green when all specifications are verified and red when verification fails. The first report shows the example above; the second shows a specification without tests.

Read specification statuses

Reports label specifications as requirements.

A specification is verified when every test marked with its ID passes. One passing test is not enough if another test for the same specification fails or skips.

When those tests have different results, the report uses the first matching status in this table, reading from top to bottom.

For example, if one test passes and another skips, the specification is SKIPPED and remains unverified. If one test fails and another skips, it is FAILED.

Missing and unfinished tests

Pytest first finds tests, then runs them. If it finds no marked tests for a declared specification, that specification is MISSING. Tests that were found but never ran are SKIPPED, including runs that only find tests or stop early. If pytest encounters an error while finding tests, such as a broken import, the verification check cannot succeed.

Tests without markers

Tests without a specification marker do not change specification statuses. They can still make pytest fail. In the duck example, the specification passes, but the unmarked cheese test fails the pytest run.

Check coverage

Coverage is the percentage of declared specifications that are verified. With no declared specifications and no errors while finding tests, the verification check succeeds and coverage is 0.0%.

Errors while finding tests make verification fail, even at 100% coverage. The report follows pytest's color settings, so --color=no disables color.

For a complete report, let pytest find the whole test suite. With -k or -m, the plugin still knows about the tests excluded by the filter and reports marked ones as DESELECTED. If you run only one folder, it cannot see tests in other folders.

Runnable examples

Start with the pizza, then try the other examples to see what makes a specification pass or remain unverified.

Run the commands below from the repository root. Dependencies are installed automatically.

Run an example

Run the Quickstart example, then replace quickstart with another example folder.

uv run --locked pytest examples/quickstart --xspec-strict

For acceptance_only, add -m xspec to run only the marked test. Without that filter, the unmarked test makes pytest fail.

Inspect the JSON report

Export the example results to see the specification status and individual test outcomes.

uv run --locked pytest examples/quickstart --xspec-json=examples/quickstart/report.json

Open examples/quickstart/report.json after running the command.

Verify all examples

Check that every example behaves as described.

uv run --locked pytest examples/test_examples.py -n 0

All five checks should pass, including checks for the examples that deliberately fail. The examples are also verified in CI.

Contributing

See CONTRIBUTING.md for setup, tests and pull requests.