TestRex

A pytest plugin that holds a test suite to one layout and one standard of test quality

My repos share one way of laying out and writing tests. This plugin carries the rules, so a repo gets them, and every fix to them, through one dependency bump instead of a copied meta suite.

It does three things:

• The meta suite. Around thirty rules that read every test file and fail on tests that cannot fail, files in the wrong place, duplicated definitions and helpers, undeclared programs, broken override signatures and pylint findings. There are no exemption lists: a test that trips a rule is fixed.
• Required programs. requires_binary (and any marker a repo adds) skips a test while collecting, before any of its fixtures start, when a program it needs is missing.
• Hermetic runs. Outbound network is closed to tests unless they are marked allow_network, and the programs a repo seals (sudo, ssh, a password manager) are refused however a test starts them.

Setting up a repo

Install it with the test dependencies. With a hash-locked requirements-dev.in:

pytest-testrex

Then opt in with a [tool.testrex] table in pyproject.toml. Without one the plugin does nothing, so having it installed never changes another project's run:

[tool.testrex]
suites = ["unit", "integration"]

and run the meta suite by subclassing it in a test module under a meta directory, by default Tests/unit/meta/:

# Tests/unit/meta/test_testrex.py
from pytest_testrex import suite


class TestRex(suite.TestRex):
    """The suite's own rules, from pytest-testrex."""

The plugin parametrizes each rule over the files it reads, so a failure names the file:

FAILED Tests/unit/meta/test_testrex.py::TestRex::test_every_test_checks_something[unit/pkg/test_mod.py]
    asserts nothing: ['test_parse (line 12)']

Layout

Each suite mirrors the repository: a test lives at the path of the code it covers.

pkg/strings.py                → Tests/unit/pkg/test_strings.py
pkg/net/fetch.py              → Tests/unit/pkg/net/test_fetch.py
scripts/setup.sh              → Tests/unit/scripts/test_setup.py
.githooks/pre-commit          → Tests/unit/githooks/test_pre_commit.py
pkg/net/fetch.py (real HTTP)  → Tests/integration/pkg/net/test_fetch.py

What decides the suite is whether a test needs a program outside the process, not whether it touches real files. A test that works in tmp_path with the code and its Python dependencies alone is a unit test; one that runs git, docker or 7zz goes in integration/ and declares it.

A test file that outgrows max_lines becomes a directory of the same name, test_<module>/, holding test_<area>.py files, a conftest.py and a <module>_helpers.py they share. Because the tree mirrors the source, two suites can hold files with the same basename, so run pytest with --import-mode=importlib.

The rules

A check counts as an assert, a mock's assert_*, pytest.fail, a pytest.raises or pytest.warns block, or a call to a helper that makes one, including helpers imported by name from a *_helpers.py beside the test or above it.

Required programs

pytestmark = pytest.mark.requires_binary("docker")       # whole module

@pytest.mark.requires_binary("gpg", "xorriso|mkisofs")   # "a|b" takes either
def test_sign_image(): ...

A repo with its own kind of requirement, such as tools it installs itself, names the marker in requirement_markers and says how to check it from its conftest.py:

def pytest_testrex_requirement_checkers(config):
    return {"requires_tool": lambda names: [f"{name} (tool)" for name in names if not installed(name)]}

Settings

Every path is relative to the directory holding pyproject.toml. A setting the plugin doesn't know, or one of the wrong type, stops the run.

Developing

python3 -m venv .venv && . .venv/bin/activate
pip install --require-hashes --no-deps -r requirements-dev.txt
pip install --no-deps --no-build-isolation -e .
pytest -q

The library is held to its own rules: Tests/unit/meta/test_testrex.py runs the suite over these tests, and Tests/unit/pytest_testrex/test_suite.py checks that a clean sample repository passes every rule and a broken one fails every rule. CI runs through NearlyTRex/Workflows at 100% line and branch coverage.

Releasing

Run Prepare Release from the Actions tab with patch, minor, major or an exact version. It opens a pull request bumping pyproject.toml; merging it tags the version, publishes a GitHub Release with the sdist and wheel, and publishes them to PyPI through trusted publishing from the pypi environment.