testudos

A testing harness for Python packages using uv's isolated environments.

Overview

Testudos runs your test suite across every Python version that is both still supported upstream and compatible with your package. It works out which versions those are by intersecting two sources:

1. Currently supported Python versions — from the endoflife.date API (v1)
2. Your package's compatibility — the requires-python field in your pyproject.toml

Cycles that endoflife.date lists but that have no stable release yet (a Python still in alpha, beta or rc) are left out unless you ask for them with --include-prerelease.

Quick start

# Install
uv tool install testudos

# See which versions would be tested, and why
testudos versions

# Run the suite on all of them
testudos run

# Run them concurrently
testudos run --parallel

# Show the exact commands without running anything
testudos run --dry-run

Installation

uv tool install testudos    # recommended
pip install testudos

Coverage needs coverage.py on your machine as well as in the test environments: testudos run --coverage combines and reports the data with it, and so do testudos coverage combine and testudos coverage report. Install the extra:

uv tool install 'testudos[coverage]'
pip install 'testudos[coverage]'

testudos runs the coverage installed alongside it, so the extra works under uv tool install, which keeps it out of your PATH. Without the extra, a coverage on PATH is used instead. Each isolated environment gets its own copy too, pinned to the same version so the data files stay readable.

Requirements

• Python 3.11 or newer
• uv
• Linux or macOS. On other platforms testudos runs, but a timeout or an interrupt stops uv and can leave the test process it started running, and testudos warns about this when a run starts.

How it works

For each Python version, testudos runs your tests in an ephemeral environment created by uv run --isolated:

uv run --isolated --python=3.12 --with pytest --directory /path/to/project pytest

• --isolated builds a throwaway environment, so nothing leaks between versions
• --python=X.Y selects the interpreter, downloading it if needed
• --with pytest puts the test runner inside that environment, where it can see your project's dependencies

--dry-run prints exactly these commands, built by the same code that runs them, so what you see is what would execute.

Test dependencies

uv run --isolated installs your project and its runtime dependencies. If your suite needs anything more — pytest plugins, fixture packages — name it, either per-invocation or in config:

testudos run --with pytest-asyncio --with pytest-mock

[tool.testudos]
test-dependencies = ["pytest-asyncio", "pytest-cov>=7"]

Usage

Choosing versions

testudos run                                # auto-detect
testudos run --python 3.11 --python 3.12    # explicit (-p)
testudos run --python 3.14.0rc1             # pre-releases are allowed explicitly
testudos run --include-prerelease           # include unreleased cycles when auto-detecting
testudos run --offline                      # skip the API, use cached or bundled data
testudos run --no-offline                   # refuse a configured offline = true
testudos run ./path/to/project              # a project other than the cwd
testudos --debug run                        # show the traceback behind a failure

testudos versions shows the same resolution as a table, marking each cycle stable or pre-release and naming any pre-release it left out.

Offline mode is also enabled by setting TESTUDOS_OFFLINE to anything except 0, false, no, off, n, f or the empty string (case-insensitive, and surrounding whitespace is ignored). Those values turn it off, overriding a configured offline = true; leaving the variable unset says nothing, so the configuration decides. --offline / --no-offline win over both.

Execution

testudos run --parallel                     # concurrent across versions (-P)
testudos run --parallel --jobs 4            # cap concurrency (-j; needs --parallel)
testudos run --no-parallel                  # override a configured parallel
testudos run --no-fail-fast                 # keep going after a failure (-F)
testudos run --timeout 300                  # seconds per version (-t)
testudos run --no-timeout                   # override a configured timeout
testudos run --verbose                      # more detail (-v)
testudos run --quiet                        # suppress failed-test output (-q)
testudos run --dry-run                      # print commands, run nothing (-n)

Ctrl-C, SIGTERM and SIGHUP all stop a run the same way: every running test process is interrupted, given five seconds to finish its teardown, and then killed. testudos exits with code 130. SIGTERM is what a cancelled CI job, docker stop or timeout(1) sends, so cancelling a pipeline does not leave suites running behind it.

Fail-fast is on by default in sequential mode. Versions it skips are reported as SKIPPED rather than dropped, so the summary always accounts for every version you asked for. It does not apply in parallel mode, where every version is already running by the time the first failure is known.

Multiple packages

testudos run --package ./pkg1 --package ./pkg2   # explicit packages (-k)
testudos run ./monorepo --discover               # every dir with a pyproject.toml (-d)
testudos run ./pkg1 --package ./pkg2 --package-jobs 2   # cap package concurrency (-J)
testudos run --package ./pkg1 --package ./pkg2 --no-parallel-packages   # one at a time

--package names the packages to test. The current directory is included only when you name it (testudos run . --package ./other), so the first example above tests exactly the two packages it lists.

Package-level and version-level concurrency are independent: --parallel / --jobs control versions within a package, --parallel-packages / --package-jobs control packages. Each package keeps its own configuration, results and coverage data.

Those two multiply, so a run is bounded by a third number that does not:

testudos run ./monorepo --discover --parallel --max-concurrency 4   # (-C)

At most --max-concurrency uv processes run at any moment, however many packages and versions there are. It defaults to one per CPU. --jobs and --package-jobs remain per-level caps within it: they can lower a level's share, never raise the total. Without it, twenty packages across four Python versions would ask for eighty simultaneous environment builds, which on a laptop or a 2-core runner shows up as timeouts and OOM kills that look like flaky tests.

A package's own max-concurrency setting applies when it is tested on its own. In a workspace the packages share one machine, so only --max-concurrency moves the limit.

Coverage

testudos run --coverage                                  # collect (-c)
testudos run --coverage --coverage-report html
testudos run --coverage --coverage-report html --coverage-report xml
testudos run --coverage --coverage-fail-under 80
testudos run --coverage --coverage-source mypackage      # default: detected
testudos run --no-coverage                               # override a configured coverage

Formats: term, term-missing, html, xml, json, lcov. Data lands in .testudos/coverage/. Without --coverage-source (or coverage-source in config), what to measure is read from the project: the packages your build backend declares, else src/, else the importable packages at the project root. A source that does not exist stops the run before any test starts, since coverage would measure nothing. The --coverage-fail-under threshold is checked against exact totals, so it gives the same verdict whichever report formats you asked for.

Coverage is reported whenever data was collected, including when a version failed its tests. The two verdicts are independent and both feed the exit code, so a coverage regression cannot hide behind an unrelated test failure. A failing version still measured the code it ran; one that fail-fast never reached measured nothing, and the report names the versions it actually covers.

Coverage data can also be managed directly:

testudos coverage combine     # merge per-version data
testudos coverage report      # generate a report (--format, --fail-under)
testudos coverage clean       # remove .testudos/coverage

Combining keeps the per-version files, so it can be repeated and .coverage.3.11 is still there afterwards to answer "which lines does 3.11 miss that 3.13 covers?". testudos coverage clean is the only command that removes coverage data.

Machine-readable output

testudos run --format json
testudos run --format json --output results.json    # (-o)
testudos run --format json --dry-run

JSON output reports per-version results plus a summary of passed, failed and skipped counts, and version_source says whether the version matrix came from endoflife.date, the cache or the bundled fallback. With --coverage it carries a coverage block (totals, threshold, report paths); in multi-package mode it has one entry per package under packages, each shaped like the single-package payload. Errors and warnings go to stderr, so stdout is always one JSON document -- including on failure, when it carries "success": false and an error.

Private indexes

testudos run --default-index https://pypi.internal.example.com/simple
testudos run --index https://extra.example.com/simple
testudos run --find-links /path/to/wheels
testudos run --no-index --find-links /path/to/wheels
testudos run --index-strategy first-index

Colour

testudos --color run      # force colour
testudos --no-color run   # disable it

These are options on testudos itself, so they go before the subcommand. NO_COLOR and FORCE_COLOR are respected too.

Configuration

Everything above can be set in [tool.testudos]:

[project]
requires-python = ">=3.11"

[tool.testudos]
# Version selection
python-versions = ["3.11", "3.12", "3.13"]   # override auto-detection
include-prerelease = false                    # include unreleased cycles
offline = false                               # never call endoflife.date

# What to run
test-command = "pytest"
test-args = ["-v", "--tb=short"]
test-dependencies = ["pytest-asyncio"]        # extra packages per isolated env
timeout = 300                                 # seconds per version

# How to run it
parallel = false
max-jobs = 4
max-concurrency = 8                           # uv processes at once (default: CPUs)

# Coverage
coverage = false
coverage-combine = true
coverage-report = ["term"]
coverage-fail-under = 80
coverage-source = ["src"]

# Package indexes
default-index = "https://pypi.internal.example.com/simple"
index = ["https://extra.example.com/simple"]
find-links = ["/path/to/wheels"]
no-index = false
index-strategy = "first-index"

CLI options win over configuration. Values are validated on load: a wrong type or an out-of-range value is an error, and an unrecognised key is a warning.

test-command is a command line, not just a program name, so a runner that needs arguments to name itself fits in one setting:

test-command = "pytest"
test-command = "python -m unittest discover"
test-command = "./run-tests"

It is split the way a shell would split it (quoted words stay together), but no shell runs it: the pieces become an argv list. Only the first word is a package name, so it is what gets installed into each isolated environment — a suite run as python -m pytest should name pytest in test-dependencies, since the interpreter is what testudos sees. A first word containing a /, like ./run-tests, is a file in your project rather than a package, so nothing is installed for it.

test-args are passed to that command exactly as written, with no filtering, so metacharacters need no escaping and no workaround:

test-args = ["--ignore-glob=*$py.class", "-W", "error:.*deprecated$:UserWarning"]

By the same token nothing in them is expanded either — --rootdir=$PWD arrives as the literal string $PWD, not as your working directory.

Documentation

• Architecture — how the pieces fit together, and the decisions behind them

Development

git clone https://github.com/martinristovski/testudos.git
cd testudos
uv sync --extra dev

just ci            # lint, type-check, test, network tests, dogfood, version check
just test          # pytest with coverage
just fix           # auto-fix lint and formatting
just dogfood       # run testudos on itself

The default test suite is hermetic: the endoflife.date API is stubbed and the version cache is redirected per test, so no test touches the network or your real ~/.cache/testudos.

Six tests do reach the network: five exercise the real uv boundary, and so resolve packages from PyPI, and one checks the canned endoflife.date response against the live API. They are marked network and deselected by default; run them with pytest -m network. CI runs them as their own step, so an index or API outage fails a step that names the cause.

License

MIT