TestRex

A pytest plugin that holds a test suite to one layout and one standard of test quality

My repos share one way of laying out and writing tests. This plugin carries the rules, so a repo gets them, and every fix to them, through one dependency bump instead of a copied meta suite.

It does five things:

• The meta suite. Around thirty rules that read every test file and fail on tests that cannot fail, files in the wrong place, duplicated definitions and helpers, undeclared programs, broken override signatures and pylint findings. There are no exemption lists: a test that trips a rule is fixed.
• Required programs. requires_binary (and any marker a repo adds) skips a test while collecting, before any of its fixtures start, when a program it needs is missing.
• Hermetic runs. Outbound network is closed to tests unless they are marked allow_network, and the programs a repo seals (sudo, ssh, a password manager) are refused however a test starts them.
• Shell scripts. Coverage for the bash scripts tests run, which coverage.py never sees, and a disposable machine to run setup scripts in as root without touching the real one.
• Library standards. Documentation and coding-standard rules for the repository's own code, each adopted with a baseline count that may only fall, so a codebase moves towards them a fix at a time.

Setting up a repo

Install it with the test dependencies. With a hash-locked requirements-dev.in:

pytest-testrex

Then opt in with a [tool.testrex] table in pyproject.toml. Without one the plugin does nothing, so having it installed never changes another project's run:

[tool.testrex]
suites = ["unit", "integration"]

and run the meta suite by subclassing it in a test module under a meta directory, by default Tests/unit/meta/:

# Tests/unit/meta/test_testrex.py
from pytest_testrex import suite


class TestRex(suite.TestRex):
    """The suite's own rules, from pytest-testrex."""

The plugin parametrizes each rule over the files it reads, so a failure names the file:

FAILED Tests/unit/meta/test_testrex.py::TestRex::test_every_test_checks_something[unit/pkg/test_mod.py]
    asserts nothing: ['test_parse (line 12)']

Layout

Each suite mirrors the repository: a test lives at the path of the code it covers.

pkg/strings.py                → Tests/unit/pkg/test_strings.py
pkg/net/fetch.py              → Tests/unit/pkg/net/test_fetch.py
scripts/setup.sh              → Tests/unit/scripts/test_setup.py
.githooks/pre-commit          → Tests/unit/githooks/test_pre_commit.py
pkg/net/fetch.py (real HTTP)  → Tests/integration/pkg/net/test_fetch.py

What decides the suite is whether a test needs a program outside the process, not whether it touches real files. A test that works in tmp_path with the code and its Python dependencies alone is a unit test; one that runs git, docker or 7zz goes in integration/ and declares it.

A test file that outgrows max_lines becomes a directory of the same name, test_<module>/, holding test_<area>.py files, a conftest.py and a <module>_helpers.py they share. Because the tree mirrors the source, two suites can hold files with the same basename, so run pytest with --import-mode=importlib.

The rules

A check counts as an assert, a mock's assert_*, pytest.fail, a pytest.raises or pytest.warns block, or a call to a helper that makes one, including helpers imported by name from a *_helpers.py beside the test or above it.

Library standards

The rules above read the tests. These read the code under library_targets, and each is off until [tool.testrex] turns it on.

A module past the limit becomes a package of the same name. Its class is assembled from mixin modules, one per area, and re-exported, so callers still from pkg.sync import Syncer and the class behaves as before:

pkg/sync.py               →  pkg/sync/__init__.py    from .local import LocalMixin
                                                     from .remote import RemoteMixin
                                                     class Syncer(LocalMixin, RemoteMixin): ...
                             pkg/sync/local.py       class LocalMixin: ...
                             pkg/sync/remote.py      class RemoteMixin: ...

Tests and pytest hooks are described by their names, so the documentation rules leave them alone. Calls are matched through imports: from subprocess import run as go; go() is subprocess.run.

[tool.testrex]
library_targets = ["Shared"]
doc_style = "comment"
doc_max_lines = 3
wrappers = { "open" = "joybox.fileops", "json.dumps" = "joybox.serialization" }
wrapper_modules = ["Shared/joybox/fileops.py", "Shared/joybox/serialization.py"]
command_paths = ["Shared/joybox/cli"]
standards = ["no-print", "no-exit", "mutable-defaults", "bare-except", "no-narration"]

[tool.testrex.baseline]
documented = 819
no-print = 40

Adopting a rule gradually. A codebase rarely meets a new rule on the day it is turned on, so each rule counts its findings across the whole library and compares the count with its baseline, zero when it has none. More findings than the baseline fail, listing them. Fewer fail too, asking for the baseline to be lowered to the new count, so a fix stays fixed and the count only falls. The baseline is a number, not a list of excused places: a new finding anywhere is a failure.

Required programs

pytestmark = pytest.mark.requires_binary("docker")       # whole module

@pytest.mark.requires_binary("gpg", "xorriso|mkisofs")   # "a|b" takes either
def test_sign_image(): ...

A repo with its own kind of requirement, such as tools it installs itself, names the marker in requirement_markers and says how to check it from its conftest.py:

def pytest_testrex_requirement_checkers(config):
    return {"requires_tool": lambda names: [f"{name} (tool)" for name in names if not installed(name)]}

Shell scripts

Both plugins load with the package and do nothing until asked.

Shell coverage measures the bash scripts the tests run, by statement, or with --shell-cov-branch also by each right-hand side of && and ||. It reads scripts with tree-sitter-bash, which the shell extra installs (pip install "pytest-testrex[shell]", or add tree-sitter and tree-sitter-bash to a hash-locked requirements file):

pytest --shell-cov-source scripts --shell-cov-source install.sh --shell-cov-fail-under 100

Every bash a test starts is traced through BASH_ENV, inherited from the environment. A test that builds its own environment takes it from the shell_traced_env fixture. Only bash is traced: sh (dash) never reads BASH_ENV.

The shell sandbox runs a script as root in a user namespace with no network, with /etc, /var, /home and the rest replaced by empty directories the test owns. PATH holds recording stubs and an allowlist of harmless tools, so a command nobody planned for fails rather than running for real:

@pytest.mark.requires_binary("bash", "unshare")
def test_enables_the_service(shell_sandbox):
    shell_sandbox.run("scripts/setup.sh", "install")
    assert shell_sandbox.ran("systemctl", "enable", "myservice")

It needs Linux with unprivileged user namespaces. Ubuntu 24.04 restricts them, so CI sets kernel.apparmor_restrict_unprivileged_userns=0 first. Where they are unavailable its tests skip, or fail with --shell-sandbox-required. Override the shell_sandbox_options fixture to add stubs (extra_stubs), tools (extra_tools), accounts and hidden directories.

Settings

Every path is relative to the directory holding pyproject.toml. A setting the plugin doesn't know, or one of the wrong type, stops the run.

Developing

python3 -m venv .venv && . .venv/bin/activate
pip install --require-hashes --no-deps -r requirements-dev.txt
pip install --no-deps --no-build-isolation -e .
pytest -q

The sandbox tests need unprivileged user namespaces and skip without them; the shell coverage tests need bash, and its cross-check against a real parser needs shfmt, which shfmt-py puts next to the venv's Python. To check any script's statements against shfmt:

python Tests/shfmt_oracle.py path/to/a.sh path/to/b.sh

The library is held to its own rules: Tests/unit/meta/test_testrex.py runs the suite over these tests, and Tests/unit/pytest_testrex/test_suite.py checks that a clean sample repository passes every rule and a broken one fails every rule. CI runs through NearlyTRex/Workflows at 100% line and branch coverage.

Releasing

Run Prepare Release from the Actions tab with patch, minor, major or an exact version. It opens a pull request bumping pyproject.toml; merging it tags the version, publishes a GitHub Release with the sdist and wheel, and publishes them to PyPI through trusted publishing from the pypi environment.