testudos

A testing harness for Python packages using uv's isolated environments.

Overview

Testudos runs your test suite across every Python version that is both still supported upstream and compatible with your package. It works out which versions those are by intersecting two sources:

1. Currently supported Python versions — from the endoflife.date API (v1)
2. Your package's compatibility — the requires-python field in your pyproject.toml

Cycles that endoflife.date lists but that have no stable release yet (a Python still in alpha, beta or rc) are left out unless you ask for them with --include-prerelease.

Quick start

# Install
uv tool install testudos

# See which versions would be tested, and why
testudos versions

# Run the suite on all of them
testudos run

# Run them concurrently
testudos run --parallel

# Show the exact commands without running anything
testudos run --dry-run

Installation

uv tool install testudos    # recommended
pip install testudos

Coverage needs nothing extra. testudos runs coverage through uv -- in each test environment, and on your machine to combine and report the data -- at one version for both, so the data files always read. It is pinned with testudos (currently 7.17.0); TESTUDOS_COVERAGE_VERSION picks another. The coverage extra still installs, and does nothing.

Requirements

• Python 3.11 or newer
• uv
• Linux or macOS. On other platforms testudos runs, but a timeout or an interrupt stops uv and can leave the test process it started running, and testudos warns about this when a run starts.

How it works

For each Python version, testudos runs your tests in an ephemeral environment created by uv run --isolated:

uv run --isolated --python=3.12 --with pytest --directory /path/to/project pytest

• --isolated builds a throwaway environment, so nothing leaks between versions
• --python=X.Y selects the interpreter, downloading it if needed
• --with pytest puts the test runner inside that environment, where it can see your project's dependencies

--dry-run prints exactly these commands, built by the same code that runs them, so what you see is what would execute.

Test dependencies

uv run --isolated installs your project, its runtime dependencies, and its dev dependency group ([dependency-groups] dev), which uv includes by default. Other groups and extras are not installed unless you name them. If your test dependencies live in a group or an extra, name that: they are then installed at the versions in uv.lock.

testudos run --group test           # [dependency-groups] test
testudos run --extra test           # [project.optional-dependencies] test

[tool.testudos]
test-groups = ["test"]
test-extras = ["test"]

([tool.uv] default-groups = ["dev", "test"] in your project has the same effect for every uv run, testudos' included.)

For anything not declared in the project, name the package itself:

testudos run --with pytest-asyncio --with pytest-mock

[tool.testudos]
test-dependencies = ["pytest-asyncio", "pytest-cov>=7"]

These are installed outside the lockfile, so a requirement without == resolves to the newest release on every run: a new plugin release can change a run with no change to your project, and a freshly published release reaches every pipeline at once. --dry-run lists them as "Unpinned --with". Prefer a group or an extra; otherwise pin them, or pass --exclude-newer 2026-10-01 (config exclude-newer) so resolution ignores anything released after that date. The same applies to a project with no uv.lock -- a Poetry or setuptools project, say -- whose whole dependency tree is resolved fresh each run, since uv does not read other tools' lock files.

The lockfile

With a uv.lock, each environment is installed from it. If pyproject.toml has changed since the lock was written, uv re-resolves in memory, says nothing and leaves uv.lock alone, so the run tests something other than what is committed. In CI you usually want that to fail:

testudos run --locked     # fail when uv.lock is out of date
testudos run --frozen     # use uv.lock as it is, without checking it

locked = true / frozen = true in [tool.testudos] do the same, and --no-locked / --no-frozen override them. UV_LOCKED=1 in the environment works too.

Usage

Choosing versions

testudos run                                # auto-detect
testudos run --python 3.11 --python 3.12    # explicit (-p)
testudos run --python 3.12.3                # a specific patch release
testudos run --python 3.14.0rc1             # pre-releases are allowed explicitly
testudos run --python 3.14t                 # the free-threaded build (or 3.14+freethreaded)
testudos run --python pypy3.10              # PyPy
testudos run --include-prerelease           # include unreleased cycles when auto-detecting
testudos run --offline                      # skip the API, use cached or bundled data
testudos run --no-offline                   # refuse a configured offline = true
testudos run ./path/to/project              # a project other than the cwd
testudos --debug run                        # show the traceback behind a failure

testudos versions shows the same resolution as a table, marking each cycle stable or pre-release and naming any pre-release it left out. For scripts and CI it also prints JSON or a plain list:

testudos versions --format json     # {"versions": [...], "version_source": "api", "supported": [...]}
testudos versions --format plain    # 3.11 3.12 3.13 3.14
testudos plan --discover             # a GitHub Actions matrix: one entry per package and version
uv python install $(testudos versions --format plain)

Free-threaded builds are where C extensions and thread-safety assumptions break first. include-variants = ["freethreaded"] adds the free-threaded build of every auto-detected cycle that has one (3.13 onwards) to the matrix; it can also be named directly, in --python, python-versions or TESTUDOS_PYTHON_VERSIONS. requires-python is matched against the language version, so 3.14t and pypy3.10 count as 3.14 and 3.10.

To hear early from the next Python, or a free-threaded build, without blocking merges on it, let it fail:

testudos run --include-prerelease --allow-failure prerelease
testudos run --allow-failure 3.15 --allow-failure freethreaded

allow-failure (config, or TESTUDOS_ALLOW_FAILURE) takes versions -- matched by cycle and build, so 3.15 covers 3.15.0b1 but not 3.15t -- and the words prerelease and freethreaded. Such a version runs and reports as usual: FAILED, ALLOWED in the table, "allowed_failure": true in the JSON, a ::warning rather than an ::error on GitHub. It does not change the exit code, and fail-fast does not stop on it.

Deciding the matrix centrally

endoflife.date moves on its own schedule, so the same commit can test a different matrix the day a Python ships or reaches end of life. An organisation that sets its own policy can name its supported cycles once, in a shared CI template or the runner environment:

export TESTUDOS_PYTHON_VERSIONS="3.11 3.12 3.13"   # stands in for endoflife.date

The list replaces endoflife.date (no network call is made) and is still intersected with each project's requires-python. A project's own python-versions, and --python, still win over it. version_source reports env. To keep endoflife.date's data but fetch it from an internal mirror of its API v1 response, set TESTUDOS_ENDOFLIFE_URL.

Refusing a guessed matrix

When neither endoflife.date nor the cache answers, testudos warns and tests its bundled list. On ephemeral CI runners there is never a cache, so a blocked or failed request takes that path. --strict-versions (config strict-versions = true, environment TESTUDOS_STRICT_VERSIONS=1) makes that an error instead.

Which interpreter

uv run --python=3.12 uses a 3.12 already installed on the machine if there is one, so the patch release tested depends on the runner image. --python-preference only-managed (config python-preference) makes uv use its own managed builds only, so every machine tests the same release.

Offline mode is also enabled by setting TESTUDOS_OFFLINE to anything except 0, false, no, off, n, f or the empty string (case-insensitive, and surrounding whitespace is ignored). Those values turn it off, overriding a configured offline = true; leaving the variable unset says nothing, so the configuration decides. --offline / --no-offline win over both.

Execution

testudos run --parallel                     # concurrent across versions (-P)
testudos run --parallel --jobs 4            # cap concurrency (-j; needs --parallel)
testudos run --no-parallel                  # override a configured parallel
testudos run --no-fail-fast                 # keep going after a failure (-F)
testudos run --timeout 300                  # seconds per version (-t; default 1800)
testudos run --no-timeout                   # no timeout at all
testudos run --no-setup                     # skip the setup phase (below)
testudos run --deadline 50m                 # the whole run's time limit
testudos run --setup-timeout 1200           # seconds per version's setup (default 900)
testudos run --verbose                      # more detail (-v)
testudos run --quiet                        # suppress failed-test output (-q)
testudos run --dry-run                      # print commands, run nothing (-n)
testudos run -- -m "not slow" -x            # extra arguments for the test command

Everything after -- goes to the test command, after the configured test-args; --test-arg is the same one argument at a time, and TESTUDOS_TEST_ARGS sets them in a shared CI template (split the way a shell would, so -m "not slow" is two arguments). The command line wins over the variable. Through the reusable workflow, end args with them:

      args: |
        --no-fail-fast
        --
        -m
        not slow

Ctrl-C, SIGTERM and SIGHUP all stop a run the same way: every running test process is interrupted, given five seconds to finish its teardown, and then killed. testudos then reports what it has -- the results table, the --format json document (with "interrupted": true), the GitHub step summary and a JUnit file per version -- with every version that had not finished marked INTERRUPTED, and exits with code 2. Coverage is not combined. SIGTERM is what a cancelled CI job, docker stop or timeout(1) sends, so cancelling a pipeline does not leave suites running behind it, and does not throw away the hour of results that came before it. While those are written, a further SIGTERM or SIGHUP is held off; a second Ctrl-C still stops at once.

Progress is a live table when stdout is a terminal. Anywhere else -- a CI log, a pipe, a file -- testudos prints one line as each version starts and finishes, and a "still running" line every minute while nothing finishes, so a long run never looks hung. FORCE_COLOR and --color add colour to those lines; they do not turn the live table on.

--deadline (or TESTUDOS_DEADLINE) bounds the whole run rather than one version: 90, 90s, 50m, 1h30m. When it passes, versions not yet started are recorded as skipped, running ones are stopped the way a timeout stops them, and the results are reported as usual -- set it a few minutes inside a CI job's own limit, which kills the job with nothing written. The reusable workflow sets it from timeout-minutes, five minutes short.

Every version runs under a timeout, 30 minutes unless you say otherwise (timeout = 0 in config, or --no-timeout, turns it off), so a hung suite stops well inside a CI job's limit rather than at it.

Before its tests, each version has a setup phase: the same uv run, with a one-line probe in place of the suite. It downloads the interpreter if needed and installs the environment into uv's cache, under its own timeout, so --timeout measures the tests and a slow first install never eats into it. The probe also reports the exact interpreter, shown in the results as 3.12 (3.12.11) and in JSON as "python" -- which patch release a cycle resolves to depends on the machine. When uv cannot provide the interpreter or install the environment, the version is reported as SETUP FAILED, with what uv said, and its tests are not run. --no-setup (config setup = false) skips the phase.

Fail-fast is on by default in sequential mode. Versions it skips are reported as SKIPPED rather than dropped, so the summary always accounts for every version you asked for. It does not apply in parallel mode, where every version is already running by the time the first failure is known.

Multiple packages

testudos run --package ./pkg1 --package ./pkg2   # explicit packages (-k)
testudos run ./monorepo --discover               # every package in the workspace (-d)
testudos run ./monorepo --discover -x 'examples/**'   # leave some out (--exclude)
testudos run ./pkg1 --package ./pkg2 --package-jobs 2   # cap package concurrency (-J)
testudos run --package ./pkg1 --package ./pkg2 --no-parallel-packages   # one at a time

--discover takes the workspace's own word for what its packages are: [tool.uv.workspace]'s members and exclude, or, for a monorepo uv does not manage, the same two globs under [tool.testudos.workspace]. Without either, every directory holding a pyproject.toml is a package except what git ignores (build output, vendored trees). --exclude adds globs, relative to the root, for what is not a package -- test fixtures, examples, project templates.

A workspace root's [tool.testudos] is its members' defaults: a member inherits every setting it does not set itself, and a setting it does set replaces the root's outright, lists included. Only a root that declares a workspace is inherited from, so an unrelated pyproject.toml further up never reaches in. --dry-run names the inherited settings.

--package names the packages to test. The current directory is included only when you name it (testudos run . --package ./other), so the first example above tests exactly the two packages it lists.

Package-level and version-level concurrency are independent: --parallel / --jobs control versions within a package, --parallel-packages / --package-jobs control packages. Each package keeps its own configuration, results and coverage data.

Those two multiply, so a run is bounded by a third number that does not:

testudos run ./monorepo --discover --parallel --max-concurrency 4   # (-C)

At most --max-concurrency uv processes run at any moment, however many packages and versions there are. It defaults to one per CPU. --jobs and --package-jobs remain per-level caps within it: they can lower a level's share, never raise the total. Without it, twenty packages across four Python versions would ask for eighty simultaneous environment builds, which on a laptop or a 2-core runner shows up as timeouts and OOM kills that look like flaky tests.

A package's own max-concurrency setting applies when it is tested on its own. In a workspace the packages share one machine, so only --max-concurrency moves the limit.

The suites share the CPUs too. pytest-xdist's -n auto sizes its pool to every CPU, so four versions on four CPUs would run sixteen busy workers. testudos sets PYTEST_XDIST_AUTO_NUM_WORKERS in each test process to the usable CPUs divided by the number of suites running at once -- in a workspace, all of them, capped by --max-concurrency -- so -n auto (and -n logical) gets a fair share. An explicit -n 8 is untouched, and so is a value you set yourself.

Coverage

testudos run --coverage                                  # collect (-c)
testudos run --coverage --coverage-report html
testudos run --coverage --coverage-report html --coverage-report xml
testudos run --coverage --coverage-fail-under 80
testudos run --coverage --coverage-source mypackage      # default: detected
testudos run --no-coverage                               # override a configured coverage

Formats: term, term-missing, html, xml, json, lcov. Data lands in .testudos/coverage/. Without --coverage-source (or coverage-source in config), what to measure is read from the project: the packages your build backend declares, else src/, else the importable packages at the project root. A source that does not exist stops the run before any test starts, since coverage would measure nothing. The --coverage-fail-under threshold is checked against exact totals, so it gives the same verdict whichever report formats you asked for.

Tests that run in subprocesses -- pytest-xdist workers, multiprocessing -- are measured too: the run is given your project's coverage configuration plus coverage's patch = ["subprocess"], written to .testudos/coverage/coveragerc (edit your project's own config, not that file). A run that measures 0% of its source says so on stderr.

Coverage is reported whenever data was collected, including when a version failed its tests. The two verdicts are independent and both feed the exit code, so a coverage regression cannot hide behind an unrelated test failure. A failing version still measured the code it ran; one that fail-fast never reached measured nothing, and the report names the versions it actually covers.

Coverage data can also be managed directly:

testudos coverage combine     # merge per-version data
testudos coverage report      # generate a report (--format, --fail-under)
testudos coverage clean       # remove .testudos/coverage

Combining keeps the per-version files, so it can be repeated and .coverage.3.11 is still there afterwards to answer "which lines does 3.11 miss that 3.13 covers?". testudos coverage clean is the only command that removes coverage data.

Machine-readable output

testudos run --format json
testudos run --format json --output results.json    # (-o)
testudos run --format json --dry-run

JSON output reports per-version results plus a summary of passed, failed and skipped counts, and version_source says whether the version matrix came from endoflife.date, the cache or the bundled fallback. With --coverage it carries a coverage block (totals, threshold, report paths); in multi-package mode it has one entry per package under packages, each shaped like the single-package payload. Errors and warnings go to stderr, so stdout is always one JSON document -- including on failure, when it carries "success": false and an error.

With --output, the document goes to the file and the terminal gets the same readable run as without --format json -- progress, the results table and the output of failed versions -- so a CI job can keep a JSON artifact and a log a person can read.

Private indexes

testudos run --default-index https://pypi.internal.example.com/simple
testudos run --index https://extra.example.com/simple
testudos run --find-links /path/to/wheels
testudos run --no-index --find-links /path/to/wheels
testudos run --index-strategy first-index

Keep credentials out of the URLs. uv reads them from the environment for an index the project names in [[tool.uv.index]] (UV_INDEX_<NAME>_USERNAME, UV_INDEX_<NAME>_PASSWORD), from .netrc, or from a keyring, and testudos passes the environment through. A URL with credentials in it still works, but anywhere testudos renders a command or a URL -- --dry-run, the JSON document, error messages -- its userinfo is written ****.

Colour

testudos --color run      # force colour
testudos --no-color run   # disable it

These are options on testudos itself, so they go before the subcommand. NO_COLOR and FORCE_COLOR are respected too.

Running in CI

GitHub Actions

The shortest setup is the reusable workflow this repository publishes:

jobs:
  test:
    uses: martinristovski/testudos/.github/workflows/testudos.yml@v0.49.0
    with:
      testudos-version: "0.49.0"
      args: --no-fail-fast --coverage --coverage-fail-under 80
      # fan-out: package-version  # one job, and one check, per package and version

It resolves the matrix with --strict-versions, uses uv-managed interpreters, enforces uv.lock when the repository has one, and installs testudos with the coverage extra. Each job keeps an artifact named <artifact-prefix>-<python> (all without per-version jobs) holding .testudos/results.json (the --format json document), the JUnit XML and any coverage reports; the artifact-prefix output names it. Inputs worth knowing:

• args: split on whitespace. For an argument with spaces or quotes, put one argument per line:

  args: |
    --no-fail-fast
    --with
    pytest-timeout >= 2.3
  

• changed-since: a git ref; only the packages a change since it reaches are tested. The checkout fetches history only when this is set (or args mentions --changed-since); otherwise it is shallow.

• name: tells this call's artifacts apart when a workflow calls the reusable workflow more than once, say once per directory. Defaults to a form of working-directory.

• python-versions: the organisation's matrix (TESTUDOS_PYTHON_VERSIONS), typically ${{ vars.PYTHON_MATRIX }} from an organisation variable.

• env (input) and env-secrets (secret): extra environment for every job, one KEY=VALUE per line; each env-secrets value is masked. A caller's own env: does not reach a reusable workflow, so this is how it gets there.

• index-name with the index-username / index-password secrets: the credentials for a private index the project names in [[tool.uv.index]], set as UV_INDEX_<NAME>_USERNAME / _PASSWORD.

• uv-version: defaults to the uv this release of the workflow was tested with; empty means the caller's [tool.uv] required-version, else the newest.

testudos' own dependencies are pinned too: each release publishes a constraints.txt (its uv.lock, with hashes) that the workflow installs with, so a new release of typer or rich cannot change what runs.

With a private index and a central matrix:

jobs:
  test:
    uses: martinristovski/testudos/.github/workflows/testudos.yml@<sha>  # v0.49.0
    with:
      testudos-version: "0.49.0"
      python-versions: ${{ vars.PYTHON_MATRIX }}
      index-name: internal
      env: |
        PYTHONWARNINGS=error::DeprecationWarning
    secrets:
      index-username: ${{ secrets.INDEX_USER }}
      index-password: ${{ secrets.INDEX_TOKEN }}

Pin the workflow by commit SHA rather than tag: a tag can be moved.

For a monorepo, fan out to one job per package and Python version, testing only what a change reaches:

    with:
      working-directory: .
      discover: true
      fan-out: package-version     # or `package`: one job per package
      # The PR's base on pull requests; the previous tip on a push.
      changed-since: ${{ github.event_name == 'pull_request' && format('origin/{0}', github.base_ref) || github.event.before }}

The plan job runs testudos plan, which selects packages exactly as a run does, and each matrix job runs testudos run <package> --python <version>. When a change reaches nothing, the test job is skipped. GitHub caps a matrix at 256 jobs; past that the plan falls back to one job per package and says so. fan-out: none (the default) is one job for everything; per-version-jobs: true still works and means fan-out: package-version.

The same in a job of your own:

test:
  runs-on: ubuntu-latest
  timeout-minutes: 60
  env:
    TESTUDOS_LOCKED: "1"                       # fail on a stale uv.lock
    TESTUDOS_PYTHON_PREFERENCE: only-managed   # the same interpreters on every runner
    TESTUDOS_STRICT_VERSIONS: "1"              # never test a guessed matrix
  steps:
    - uses: actions/checkout@<sha>
    - uses: astral-sh/setup-uv@<sha>
      with: { enable-cache: true }
    - run: uvx testudos@0.49.0 run --parallel --no-fail-fast --junit-xml .testudos/junit

Or fan out in a workflow of your own, one job per package and Python version, so each is its own check and can be re-run on its own:

plan:
  runs-on: ubuntu-latest
  outputs: { matrix: "${{ steps.p.outputs.matrix }}" }
  steps:
    - uses: actions/checkout@<sha>
      with: { fetch-depth: 0 }                 # for --changed-since
    - uses: astral-sh/setup-uv@<sha>
    - id: p
      run: echo "matrix=$(uvx testudos@0.49.0 plan --discover --strict-versions --changed-since origin/main)" >> "$GITHUB_OUTPUT"
test:
  needs: plan
  if: fromJSON(needs.plan.outputs.matrix).include[0] != null   # nothing to test
  runs-on: ubuntu-latest
  strategy:
    fail-fast: false
    matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
  name: Test (${{ matrix.package }}, Python ${{ matrix.python }})
  steps:
    - uses: actions/checkout@<sha>
    - uses: astral-sh/setup-uv@<sha>
      with: { enable-cache: true }
    - run: uvx testudos@0.49.0 run ${{ matrix.path }} --python ${{ matrix.python }} --locked

testudos plan prints {"include": [{"package", "path", "python", "id"}, ...]} -- --matrix package makes it one entry per package, with python: all -- and --format json adds the packages, their versions and what --changed-since left out.

Whatever the CI system:

• Pin testudos (uvx testudos@0.49.0): every change merged here is published, so an unpinned install changes under you. Pin its dependencies as well, with the constraints file the release publishes: uvx --constraints https://github.com/martinristovski/testudos/releases/download/v0.49.0/constraints.txt --from 'testudos==0.49.0' testudos.
• Pin uv (astral-sh/setup-uv with version:, or [tool.uv] required-version): uv is still 0.x, and its behaviour changes between releases.
• --no-fail-fast, so one run reports every failing version.
• --locked when uv.lock is committed, and test dependencies in a dependency group or extra (--group test) rather than --with, so they come from the lockfile.
• --strict-versions, or a central TESTUDOS_PYTHON_VERSIONS, so the matrix is never the bundled guess.
• --python-preference only-managed for the same interpreter everywhere.
• The default 30-minute timeout, or a --timeout sized for your suite; the setup phase keeps interpreter downloads out of it.
• --format json --output results.json for a machine-readable record next to the readable log, and the exit code (below) to tell a failing suite from an outage.

On GitHub Actions (GITHUB_ACTIONS=true, which the runner sets) testudos also writes what GitHub reads from a job:

• each version's output inside a ::group::, so the log folds to one line per version, and inside a ::stop-commands:: block with a random token, so a line the suite prints that starts with :: -- a fake ::error, an ::add-mask:: -- is shown as text rather than obeyed;
• with --junit-xml, an ::error annotation for each failing test, on its line in the pull request's diff and naming every version it fails on (tests.test_x::test_parse -- fails on 3.12, 3.13): a test that fails everywhere is one annotation, not one per version;
• an ::error annotation for each failed version the tests do not account for -- a setup failure, a timeout, a suite that is not pytest -- and for a missed coverage threshold, and a ::warning for each warning (a fallback matrix, an unknown config key). These show on the run's summary page and the pull request's checks. GitHub shows at most ten error annotations per step, so the tests failing on the most versions come first and the last annotation says how many more there are;
• a results table appended to $GITHUB_STEP_SUMMARY, with a grid of the failing tests by Python version (❌ failed, ✅ passed, – not run).

--ci-format github turns this on anywhere, --ci-format none turns it off.

testudos run --junit-xml .testudos/junit     # a JUnit XML report per version

For pytest, each version writes .testudos/junit/<version>.xml itself, with a suite named python-<version> -- <package>/python-<version> when the run has more than one package, so a report viewer reading every file keeps them apart -- which test-report actions and GitLab or Jenkins read. A version pytest did not write one for -- its setup failed, it timed out, fail-fast skipped it, or the suite is not pytest -- gets a one-test-case report from testudos, so no version is missing. A relative directory is inside each package; an absolute one gets a subdirectory per package, named after the package (two packages can share a directory name, never a package name). The paths are in the JSON as "junit_xml".

In a monorepo, test only what a change reaches:

testudos run --discover --changed-since origin/main

A package is tested when a file inside it changed since the branch left origin/main (committed, uncommitted or untracked), or when it depends on one that did -- through [project].dependencies, optional dependencies, dependency groups or [tool.uv.sources] -- directly or through other packages. A change outside every package (CI configuration, say) tests all of them, and so does a change a package owns that none of the packages nested inside it does: when the repository root is itself a package, its uv.lock or conftest.py reaches every member. The rest are listed as not affected, and as "not_affected" in JSON, with the reason each tested package was picked in "affected_by" (and in --verbose output); with nothing affected the run exits 0.

Rules in the repository root's pyproject.toml refine that, as globs written the way .gitignore writes them:

[tool.testudos.changed-since]
ignore = ["*.md", "docs/**"]        # these changes affect no package
global = ["libs/shared/fixtures/**"] # these affect every package
``` It needs the ref's history:
`actions/checkout` with `fetch-depth: 0`. `github.base_ref` is empty on a
`push`, so a workflow that runs on both compares a push against
`github.event.before`, the commit the branch pointed at before it.

### Exit codes

| Code | Meaning |
|---|---|
| 0 | Every version passed, and coverage met its threshold |
| 1 | Tests failed, an environment could not be set up (a stale lock under `--locked`, requirements that do not resolve, a build that fails), or coverage missed `--coverage-fail-under` |
| 2 | Interrupted: Ctrl-C, `SIGTERM` or `SIGHUP` |
| 3 | testudos could not run: uv or coverage missing, no version data under `--strict-versions`, an unwritable `--output`, or testudos itself crashed |
| 4 | The command line or a configuration is wrong: an unknown option or value, an invalid `[tool.testudos]`, a `requires-python` nothing satisfies, a missing coverage source |

CI can retry on 3, send 4 to whoever owns the configuration, and treat 1 as
the result. A setup failure is a 1 rather than a 3 because retrying it almost
never helps: the lockfile, the requirements or the build is what needs fixing.

### Environment variables

Most `run` options can be set from the environment, which is how shared CI
templates are usually configured; a flag on the command line still wins.
`testudos run --help` lists them: `TESTUDOS_TIMEOUT`, `TESTUDOS_PARALLEL`,
`TESTUDOS_JOBS`, `TESTUDOS_MAX_CONCURRENCY`, `TESTUDOS_FAIL_FAST`,
`TESTUDOS_COVERAGE`, `TESTUDOS_COVERAGE_REPORT`, `TESTUDOS_COVERAGE_FAIL_UNDER`,
`TESTUDOS_WITH`, `TESTUDOS_GROUP`, `TESTUDOS_EXTRA`, `TESTUDOS_LOCKED`,
`TESTUDOS_FROZEN`, `TESTUDOS_EXCLUDE_NEWER`, `TESTUDOS_PYTHON_PREFERENCE`,
`TESTUDOS_SETUP`, `TESTUDOS_SETUP_TIMEOUT`, `TESTUDOS_DEFAULT_INDEX`,
`TESTUDOS_INDEX`, `TESTUDOS_TEST_ARGS`, `TESTUDOS_ALLOW_FAILURE`, `TESTUDOS_COVERAGE_VERSION`, `TESTUDOS_DEADLINE`, `TESTUDOS_FORMAT`, `TESTUDOS_OUTPUT`, `TESTUDOS_PACKAGE_JOBS`.
Repeatable options take a space-separated list (`TESTUDOS_GROUP="test lint"`).
`TESTUDOS_OFFLINE`, `TESTUDOS_STRICT_VERSIONS`, `TESTUDOS_PYTHON_VERSIONS`,
`TESTUDOS_ENDOFLIFE_URL` and `TESTUDOS_TRACEBACK` are described with their
features above.

### Proxies and certificates

testudos makes one HTTP request, to endoflife.date, with Python's standard
library: it honours `HTTPS_PROXY` and `NO_PROXY` and verifies TLS against the
operating system's trust store, so a TLS-inspecting corporate proxy whose
certificate the machine trusts works without extra configuration. uv has its
own settings for the package index (`UV_NATIVE_TLS` / `UV_SYSTEM_CERTS`).

## Configuration

Everything above can be set in `[tool.testudos]`:

```toml
[project]
requires-python = ">=3.11"

[tool.testudos]
# Version selection
python-versions = ["3.11", "3.12", "3.13"]   # override auto-detection
include-prerelease = false                    # include unreleased cycles
include-variants = ["freethreaded"]           # add 3.13t, 3.14t, ... to auto-detection
allow-failure = ["prerelease", "freethreaded"] # report these, but do not fail on them
offline = false                               # never call endoflife.date
strict-versions = false                       # fail rather than test the bundled list
python-preference = "only-managed"            # uv-managed interpreters only

# What to run
test-command = "pytest"
test-args = ["-v", "--tb=short"]
test-groups = ["test"]                        # [dependency-groups] to install
test-extras = ["test"]                        # extras to install
test-dependencies = ["pytest-asyncio"]        # extra packages per isolated env
timeout = 300                                 # seconds per version (0: none)
setup = true                                  # install each environment first
setup-timeout = 900                           # seconds per version's setup

# The lockfile
locked = true                                 # fail when uv.lock is stale
frozen = false                                # use uv.lock without checking it
exclude-newer = "2026-10-01"                  # ignore releases after this date

# How to run it
junit-xml = ".testudos/junit"                 # a JUnit XML report per version
parallel = false
max-jobs = 4
max-concurrency = 8                           # uv processes at once (default: CPUs)

# Coverage
coverage = false
coverage-combine = true
coverage-report = ["term"]
coverage-fail-under = 80
coverage-source = ["src"]

# Package indexes
default-index = "https://pypi.internal.example.com/simple"
index = ["https://extra.example.com/simple"]
find-links = ["/path/to/wheels"]
no-index = false
index-strategy = "first-index"

# --changed-since rules (read from the repository root's pyproject.toml)
# The workspace's packages, for a monorepo uv does not manage
[tool.testudos.workspace]
members = ["libs/*", "apps/*"]
exclude = ["libs/legacy"]

[tool.testudos.changed-since]
ignore = ["*.md", "docs/**"]                  # changes that affect no package
global = []                                   # changes that affect every package

CLI options win over configuration. Values are validated on load: a wrong type or an out-of-range value is an error, and an unrecognised key is a warning.

test-command is a command line, not just a program name, so a runner that needs arguments to name itself fits in one setting:

test-command = "pytest"
test-command = "python -m unittest discover"
test-command = "./run-tests"

It is split the way a shell would split it (quoted words stay together), but no shell runs it: the pieces become an argv list. Only the first word is a package name, so it is what gets installed into each isolated environment — a suite run as python -m pytest should name pytest in test-dependencies, since the interpreter is what testudos sees. A first word containing a /, like ./run-tests, is a file in your project rather than a package, so nothing is installed for it.

test-args are passed to that command exactly as written, with no filtering, so metacharacters need no escaping and no workaround:

test-args = ["--ignore-glob=*$py.class", "-W", "error:.*deprecated$:UserWarning"]

By the same token nothing in them is expanded either — --rootdir=$PWD arrives as the literal string $PWD, not as your working directory.

Documentation

• Architecture — how the pieces fit together, and the decisions behind them

Development

git clone https://github.com/martinristovski/testudos.git
cd testudos
uv sync --extra dev

just ci            # lint, type-check, test, network tests, dogfood, version check
just test          # pytest with coverage
just fix           # auto-fix lint and formatting
just dogfood       # run testudos on itself

The default test suite is hermetic: the endoflife.date API is stubbed and the version cache is redirected per test, so no test touches the network or your real ~/.cache/testudos.

Seven tests do reach the network: six exercise the real uv boundary, and so resolve packages from PyPI, and one checks the canned endoflife.date response against the live API. They are marked network and deselected by default; run them with pytest -m network. CI runs them as their own step, so an index or API outage fails a step that names the cause.

License

MIT