testudos
A testing harness for Python packages using uv's isolated environments.
Overview
Testudos runs your test suite across every Python version that is both still supported upstream and compatible with your package. It works out which versions those are by intersecting two sources:
- Currently supported Python versions — from the endoflife.date API (v1)
- Your package's compatibility — the
requires-pythonfield in yourpyproject.toml
Cycles that endoflife.date lists but that have no stable release yet (a
Python still in alpha, beta or rc) are left out unless you ask for them with
--include-prerelease.
Quick start
# Install
uv tool install testudos
# See which versions would be tested, and why
testudos versions
# Run the suite on all of them
testudos run
# Run them concurrently
testudos run --parallel
# Show the exact commands without running anything
testudos run --dry-run
Installation
uv tool install testudos # recommended
pip install testudos
Coverage needs nothing extra. testudos runs coverage through uv -- in each
test environment, and on your machine to combine and report the data -- at one
version for both, so the data files always read. It is pinned with testudos
(currently 7.17.0); TESTUDOS_COVERAGE_VERSION picks another. The coverage
extra still installs, and does nothing.
Requirements
- Python 3.11 or newer
- uv 0.4.27 or newer (the first with
uv run --group). An older uv stops the run with exit 3 and says so, rather than failing every version's setup. - Linux or macOS. On other platforms testudos runs, but a timeout or an
interrupt stops
uvand can leave the test process it started running, and testudos warns about this when a run starts.
How it works
For each Python version, testudos runs your tests in an ephemeral environment
created by uv run --isolated:
uv run --isolated --python=3.12 --with pytest --directory /path/to/project pytest
--isolatedbuilds a throwaway environment, so nothing leaks between versions--python=X.Yselects the interpreter, downloading it if needed--with pytestputs the test runner inside that environment, where it can see your project's dependencies
--dry-run prints exactly these commands, built by the same code that runs
them, so what you see is what would execute.
Test dependencies
uv run --isolated installs your project, its runtime dependencies, and its
dev dependency group ([dependency-groups] dev), which uv includes by
default. Other groups and extras are not installed unless you name them. If
your test dependencies live in a group or an extra, name that: they are then
installed at the versions in uv.lock.
testudos run --group test # [dependency-groups] test
testudos run --extra test # [project.optional-dependencies] test
[tool.testudos]
test-groups = ["test"]
test-extras = ["test"]
([tool.uv] default-groups = ["dev", "test"] in your project has the same
effect for every uv run, testudos' included.)
For anything not declared in the project, name the package itself:
testudos run --with pytest-asyncio --with pytest-mock
[tool.testudos]
test-dependencies = ["pytest-asyncio", "pytest-cov>=7"]
These are installed outside the lockfile, so a requirement without ==
resolves to the newest release on every run: a new plugin release can change a
run with no change to your project, and a freshly published release reaches
every pipeline at once. A run warns about them before it starts, the JSON
lists them as unpinned_dependencies, and --dry-run as "Unpinned --with". A
package uv.lock pins is not counted: the environment already has it at its
locked version. --require-pinned (require-pinned = true,
TESTUDOS_REQUIRE_PINNED=1) refuses to run with any, exit 4. Prefer a
group or an extra; otherwise pin them, or pass --exclude-newer 2026-10-01
(config exclude-newer) so resolution ignores anything released after that
date. The same applies to a project with no uv.lock -- a Poetry or setuptools
project, say -- whose whole dependency tree is resolved fresh each run, since uv
does not read other tools' lock files.
The lockfile
With a uv.lock, each environment is installed from it. If pyproject.toml
has changed since the lock was written, uv re-resolves in memory, says nothing
and leaves uv.lock alone, so the run tests something other than what is
committed. In CI you usually want that to fail:
testudos run --locked # fail when uv.lock is out of date
testudos run --frozen # use uv.lock as it is, without checking it
locked = true / frozen = true in [tool.testudos] do the same, and
--no-locked / --no-frozen override them. UV_LOCKED=1 in the environment
works too.
Other dependency versions
The lockfile is one answer to "which versions of my dependencies?". A library promises a range, and two other points of it are worth testing -- on a nightly or pre-release job rather than every pull request:
testudos run --resolution lowest-direct # every direct requirement at its declared floor
testudos run --resolution highest # the newest releases, ignoring uv.lock
testudos run --no-sources # published metadata, not [tool.uv.sources]
lowest-direct catches a floor that is too low -- requests>=2.20 while
using an API added in 2.28 -- before a consumer with an older lock does.
highest catches an upstream release that breaks you before your consumers
resolve to it. --no-sources tests a workspace member against the releases its
[project].dependencies allow rather than its siblings' working trees, which
is what it will get once published; it pairs well with lowest-direct.
All three resolve anew, outside uv.lock (which is never written), so they
cannot be combined with --locked / --frozen; a configured locked = true
gives way to them. lowest-direct applies to --with requirements too, the
test runner included: an unbounded pytest would be pytest's first release, so
a run warns about any requirement nothing gives a floor -- give it one
(test-dependencies = ["pytest>=8"]) or install it from a dependency group
that has one. The dry run and its JSON say which resolution a run uses;
resolution and no-sources in [tool.testudos], and TESTUDOS_RESOLUTION /
TESTUDOS_NO_SOURCES, set them too.
The installed package
uv installs the project editable, so the suite imports it from the source tree,
and a wheel that leaves a module or a data file out still passes. tox catches
that by building the package first; --no-editable does the same:
testudos run --no-editable # test the project installed as a built wheel
(editable = false in [tool.testudos], or TESTUDOS_EDITABLE=0.) Coverage
works with it: the code under test ran from a temporary environment's
site-packages, so it is measured by import name and the rcfile testudos writes
maps */site-packages/ back to the source tree when the data is combined --
always, a single version included. A src/ layout makes this airtight; with
the package at the project root, a conftest.py there can put the root on
sys.path and import the source tree after all.
Usage
Choosing versions
testudos run # auto-detect
testudos run --python 3.11 --python 3.12 # explicit (-p)
testudos run --python 3.12.3 # a specific patch release
testudos run --python 3.14.0rc1 # pre-releases are allowed explicitly
testudos run --python 3.14t # the free-threaded build (or 3.14+freethreaded)
testudos run --python pypy3.10 # PyPy
testudos run --include-prerelease # include unreleased cycles when auto-detecting
testudos run --offline # skip the API, use cached or bundled data
testudos run --no-offline # refuse a configured offline = true
testudos run ./path/to/project # a project other than the cwd
testudos --debug run # show the traceback behind a failure
testudos versions shows the same resolution as a table, marking each cycle
stable or pre-release and naming any pre-release it left out. For scripts and
CI it also prints JSON or a plain list:
testudos versions --format json # {"versions": [...], "version_source": "api", "supported": [...]}
testudos versions --format plain # 3.11 3.12 3.13 3.14
testudos run --discover --matrix package-version # a GitHub Actions matrix, running nothing
uv python install $(testudos versions --format plain)
Free-threaded builds are where C extensions and thread-safety assumptions
break first. include-variants = ["freethreaded"] adds the free-threaded build
of every auto-detected cycle that has one (3.13 onwards) to the matrix; it can
also be named directly, in --python, python-versions or
TESTUDOS_PYTHON_VERSIONS. requires-python is matched against the language
version, so 3.14t and pypy3.10 count as 3.14 and 3.10.
To hear early from the next Python, or a free-threaded build, without blocking merges on it, let it fail:
testudos run --include-prerelease --allow-failure prerelease
testudos run --allow-failure 3.15 --allow-failure freethreaded
allow-failure (config, or TESTUDOS_ALLOW_FAILURE) takes versions --
matched by cycle and build, so 3.15 covers 3.15.0b1 but not 3.15t --
and the words prerelease and freethreaded. Such a version runs and reports
as usual: FAILED, ALLOWED in the table, "allowed_failure": true in the
JSON, a ::warning rather than an ::error on GitHub. It does not change the
exit code, and fail-fast does not stop on it.
Deciding the matrix centrally
endoflife.date moves on its own schedule, so the same commit can test a different matrix the day a Python ships or reaches end of life. An organisation that sets its own policy can name its supported cycles once, in a shared CI template or the runner environment:
export TESTUDOS_PYTHON_VERSIONS="3.11 3.12 3.13" # stands in for endoflife.date
The list replaces endoflife.date (no network call is made) and is still
intersected with each project's requires-python. A project's own
python-versions, and --python, still win over it. version_source reports
env. To keep endoflife.date's data but fetch it from an internal mirror of
its API v1 response, set TESTUDOS_ENDOFLIFE_URL.
A floor below the matrix
Auto-detection tests only supported cycles, so a requires-python that admits
end-of-life ones -- >=3.9 when 3.11 is the oldest supported -- tells
installers the package works on Pythons nothing runs it on. testudos says so:
Warning: requires-python '>=3.9' admits 3.9, 3.10, past end of life, which
auto-detection does not test. Raise the floor, or name them in python-versions...
untested-floor in [tool.testudos] (or TESTUDOS_UNTESTED_FLOOR, an
organisation's default that a project's own setting overrides) is warn, the
default, ignore, or error, which stops the run with exit 4.
testudos versions --format json lists them as untested_admitted. Named
versions (--python, python-versions) are a deliberate choice and are not
checked.
Refusing a guessed matrix
When neither endoflife.date nor the cache answers, testudos warns and tests its
bundled list. On ephemeral CI runners there is never a cache, so a blocked or
failed request takes that path. --strict-versions (config
strict-versions = true, environment TESTUDOS_STRICT_VERSIONS=1) makes that
an error instead.
Which interpreter
uv run --python=3.12 uses a 3.12 already installed on the machine if there is
one, so the patch release tested depends on the runner image.
--python-preference only-managed (config python-preference) makes uv use
its own managed builds only, so every machine tests the same release.
When requires-python rules out some releases of a cycle -- >=3.12.4 --
testudos asks uv for a release it allows (--python=>=3.12.4,==3.12.*) rather
than for the cycle, so a machine whose own 3.12 is older (Ubuntu 24.04 ships
3.12.3) gets a managed 3.12 instead of a failed setup. A patch release you name
(--python 3.12.3) is asked for as written.
Offline mode is also enabled by setting TESTUDOS_OFFLINE to anything except
0, false, no, off, n, f or the empty string (case-insensitive, and
surrounding whitespace is ignored). Those values turn it off, overriding a
configured offline = true; leaving the variable unset says nothing, so the
configuration decides. --offline / --no-offline win over both.
Execution
testudos run --parallel # concurrent across versions (-P)
testudos run --parallel --jobs 4 # cap concurrency (-j; needs --parallel)
testudos run --no-parallel # override a configured parallel
testudos run --no-fail-fast # keep going after a failure (-F)
testudos run --timeout 300 # seconds per version (-t; default 1800)
testudos run --no-timeout # no timeout at all
testudos run --no-setup # skip the setup phase (below)
testudos run --deadline 50m # the whole run's time limit
testudos run --setup-timeout 1200 # seconds per version's setup (default 900)
testudos run --verbose # more detail (-v)
testudos run --quiet # suppress failed-test output (-q)
testudos run --dry-run # print commands, run nothing (-n)
testudos run -- -m "not slow" -x # extra arguments for the test command
Everything after -- goes to the test command, after the configured
test-args; --test-arg is the same one argument at a time, and
TESTUDOS_TEST_ARGS sets them in a shared CI template (split the way a shell
would, so -m "not slow" is two arguments). The command line wins over the
variable.
Ctrl-C, SIGTERM and SIGHUP all stop a run the same way: every running
test process is interrupted, given five seconds to finish its teardown, and
then killed. testudos then reports what it has -- the results table, the
--format json document (with "interrupted": true), the GitHub step summary
and a JUnit file per version -- with every version that had not finished marked
INTERRUPTED, and exits with code 2. Coverage is not combined. SIGTERM is
what a cancelled CI job, docker stop or timeout(1) sends, so cancelling a
pipeline does not leave suites running behind it, and does not throw away the
hour of results that came before it. While those are written, a further
SIGTERM or SIGHUP is held off; a second Ctrl-C still stops at once.
Progress is a live table when stdout is a terminal. Anywhere else -- a CI
log, a pipe, a file -- testudos prints one line as each version starts and
finishes, and a "still running" line every minute while nothing finishes, so a
long run never looks hung. FORCE_COLOR and --color add colour to those
lines; they do not turn the live table on.
--deadline (or TESTUDOS_DEADLINE) bounds the whole run rather than one
version: 90, 90s, 50m, 1h30m. When it passes, versions not yet started
are recorded as skipped, running ones are stopped the way a timeout stops them,
and the results are reported as usual -- set it a few minutes inside a CI job's
own limit, which kills the job with nothing written. Reporting what ran gets two
minutes past the deadline: coverage still reported for a run the deadline cut
short, but no longer than that -- a package whose coverage is not reported by
then fails it (a threshold nobody checked has not passed), and the JUnit
reports, step summary and JSON are still written. So leave the job at least
that much more than the deadline. An interrupt while results are being
reported ends only the step it arrives in: the rest are written, and the run
exits 2.
A version is over when its tests are. A process the suite started and never
stopped -- a server a fixture left running -- is stopped once the tests have
exited, rather than waited for, and the run warns naming the version
("stray_processes": true in JSON). The version's verdict is the suite's own
exit code.
Every version runs under a timeout, 30 minutes unless you say otherwise
(timeout = 0 in config, or --no-timeout, turns it off), so a hung suite
stops well inside a CI job's limit rather than at it.
Before its tests, each version has a setup phase: the same uv run, with a
one-line probe in place of the suite. It downloads the interpreter if needed
and installs the environment into uv's cache, under its own timeout, so
--timeout measures the tests and a slow first install never eats into it.
The tests then run uv run --offline, from that cache: they reach no index,
and get none of uv's index credentials (see Private indexes).
Setup checks that they can, by running its probe once more offline; where the
cache cannot rebuild the environment -- uv keeps none (UV_NO_CACHE), or a
requirement cannot be resolved offline -- that version's tests run online,
with a warning and "ran_online": true in JSON.
The probe also reports the exact interpreter, shown in the results as
3.12 (3.12.11) and in JSON as "python" -- which patch release a cycle
resolves to depends on the machine. When uv cannot provide the interpreter or
install the environment, the version is reported as SETUP FAILED, with what
uv said, and its tests are not run. --no-setup (config setup = false)
skips the phase.
Fail-fast is on by default in sequential mode. Versions it skips are reported
as SKIPPED rather than dropped, so the summary always accounts for every
version you asked for. It does not apply in parallel mode, where every version
is already running by the time the first failure is known.
Multiple packages
testudos run --package ./pkg1 --package ./pkg2 # explicit packages (-k)
testudos run ./monorepo --discover # every package in the workspace (-d)
testudos run ./monorepo --discover -x 'examples/**' # leave some out (--exclude)
--discover takes the workspace's own word for what its packages are:
[tool.uv.workspace]'s members and exclude, or, for a monorepo uv does not
manage, the same two globs under [tool.testudos.workspace]. Without either,
every directory holding a pyproject.toml is a package except what git ignores
(build output, vendored trees). --exclude adds globs, relative to the root,
for what is not a package -- test fixtures, examples, project templates.
A workspace root's [tool.testudos] is its members' defaults: a member
inherits every setting it does not set itself, and a setting it does set
replaces the root's outright, lists included. Only a root that declares a
workspace is inherited from, so an unrelated pyproject.toml further up never
reaches in. --dry-run names the inherited settings.
--package names the packages to test. The current directory is included
only when you name it (testudos run . --package ./other), so the first
example above tests exactly the two packages it lists.
Packages run alongside each other, and --parallel / --jobs run each
package's versions alongside each other too. Each package keeps its own
configuration, results and coverage data.
Those multiply, so a run is bounded by one number that does not:
testudos run ./monorepo --discover --parallel --max-concurrency 4 # (-C)
At most --max-concurrency uv processes run at any moment, however many
packages and versions there are. It defaults to one per CPU; -C 1 tests
one version of one package at a time. --jobs remains a per-package cap
within it: it can lower a package's share, never raise the total. Without it,
twenty packages across four Python versions would ask for eighty simultaneous
environment builds, which on a laptop or a 2-core runner shows up as timeouts
and OOM kills that look like flaky tests.
Suites that share something -- one database, a fixed port, a fixture
directory -- must not overlap, across packages or across one package's
versions alike: run them with -C 1.
A package's own max-concurrency setting applies when it is tested on its
own. In a workspace the packages share one machine, so only --max-concurrency
moves the limit.
The suites share the CPUs too. pytest-xdist's -n auto sizes its pool to every
CPU, so four versions on four CPUs would run sixteen busy workers. testudos sets
PYTEST_XDIST_AUTO_NUM_WORKERS in each test process to the usable CPUs divided
by the number of suites running at once -- in a workspace, all of them, capped
by --max-concurrency -- so -n auto (and -n logical) gets a fair share. An
explicit -n 8 is untouched, and so is a value you set yourself.
Coverage
testudos run --coverage # collect (-c)
testudos run --coverage --coverage-report html
testudos run --coverage --coverage-report html --coverage-report xml
testudos run --coverage --coverage-fail-under 80
testudos run --coverage --coverage-source mypackage # default: detected
testudos run --no-coverage # override a configured coverage
Formats: term, term-missing, html, xml, json, lcov. Data lands in
.testudos/coverage/. .testudos/ holds what a run writes and ignores itself,
with a .gitignore of its own, so it stays out of git status and of
--changed-since. Without --coverage-source (or coverage-source in
config), what to measure is read from the project: the packages your build
backend declares, else src/, else the importable packages at the project
root. A source that does not exist stops the run before any test starts, since
coverage would measure nothing. The --coverage-fail-under threshold is checked
against exact totals, so it gives the same verdict whichever report formats
you asked for.
Tests that run in subprocesses -- pytest-xdist workers, multiprocessing --
are measured too: the run is given your project's coverage configuration plus
coverage's patch = ["subprocess"], written to .testudos/coverage/coveragerc
(edit your project's own config, not that file). A run that measures 0% of its
source says so on stderr.
Coverage is reported whenever data was collected, including when a version failed its tests. The two verdicts are independent and both feed the exit code, so a coverage regression cannot hide behind an unrelated test failure. A failing version still measured the code it ran; one that fail-fast never reached measured nothing, and the report names the versions it actually covers.
Coverage data can also be managed directly:
testudos coverage combine # merge per-version data
testudos coverage report # generate a report (--format; --fail-under, else the configured one)
testudos coverage clean # remove .testudos/coverage
Combining keeps the per-version files, so it can be repeated and
.coverage.3.11 is still there afterwards to answer "which lines does 3.11
miss that 3.13 covers?". testudos coverage clean is the only command that
removes coverage data.
Machine-readable output
testudos run --format json
testudos run --format json --output results.json # (-o)
testudos run --format json --dry-run
JSON output reports per-version results plus a summary of passed, failed and
skipped counts, and version_source says whether the version matrix came from
endoflife.date, the cache or the bundled fallback. With --coverage it carries
a coverage block (totals, threshold, report paths); in multi-package mode it
has one entry per package under packages, each shaped like the
single-package payload. Errors and warnings go to stderr, so stdout is always
one JSON document -- including on failure, when it carries "success": false
and an error.
With --output, the document goes to the file and the terminal gets the same
readable run as without --format json -- progress, the results table and the
output of failed versions -- so a CI job can keep a JSON artifact and a log a
person can read.
Private indexes
testudos run --default-index https://pypi.internal.example.com/simple
testudos run --index https://extra.example.com/simple
testudos run --find-links /path/to/wheels
testudos run --no-index --find-links /path/to/wheels
testudos run --index-strategy first-index
These reach every uv command a run makes: each test environment, and the
coverage testudos runs on the host to read your coverage configuration, combine
and report -- so with --coverage nothing goes to PyPI behind your back. The
[tool.testudos] keys apply to testudos coverage combine and report too.
exclude-newer travels the same way.
Keep credentials out of the URLs. uv reads them from the environment for an
index the project names in [[tool.uv.index]] (UV_INDEX_<NAME>_USERNAME,
UV_INDEX_<NAME>_PASSWORD), from .netrc, or from a keyring, and the setup
phase gets the environment as it is. A URL with credentials in it still works,
but anywhere testudos renders a command or a URL -- --dry-run, the JSON
document, error messages -- its userinfo is written ****.
The suite never sees those variables. After setup, the tests run with uv run --offline from what setup put in uv's cache, so uv needs no credentials, and
testudos leaves UV_INDEX_<NAME>_USERNAME, UV_INDEX_<NAME>_PASSWORD and
UV_PUBLISH_* out of the test process's environment. A suite that needs a
credential itself gets it under a name of its own (TEST_INDEX_TOKEN).
The tests run online, with the credentials, when uv's cache cannot rebuild
the environment offline -- no cache to keep (UV_NO_CACHE, or no-cache in
[tool.uv]), or a requirement uv cannot resolve offline -- and testudos warns
naming the version ("ran_online": true in JSON). Under --no-setup they
always run online with the credentials, and ran_online is not set: nothing
fell back. Keep a uv cache, with setup on, if you want the credentials kept
from the suite.
This keeps a token from being exposed by accident -- a test, a plugin or a
crash report that dumps os.environ. It is not a defence against a hostile
pull request: setup installs the project, which runs the pull request's build
code (a setup.py, a hatch build hook, whatever [build-system] names) with
the credentials in its environment. Nor does it cover credentials written into
a URL (uv's cache is keyed on the whole URL, so they stay), or .netrc or a
keyring, which any process can read. A job that tests pull requests from
outside the organisation should hold no credential that matters: a read-only,
short-lived token, or none -- a token that can read a private index can often
publish to it.
Colour
testudos --color run # force colour
testudos --no-color run # disable it
These are options on testudos itself, so they go before the subcommand.
NO_COLOR and FORCE_COLOR are respected too.
Running in CI
GitHub Actions
testudos is a command, so a job is uv plus one step. Pin each action by commit
SHA, and testudos and uv by version. The examples leave the pins to you:
<sha> is the action's commit, <version> the testudos release (the newest
on PyPI), <upload-time> a timestamp
after that release's upload, such as 2026-11-08T00:00:00Z, and
<uv-version> a uv release.
test:
runs-on: ubuntu-latest
timeout-minutes: 60
permissions: { contents: read } # the suite is code from the pull request
env:
TESTUDOS_DEADLINE: 55m # stop and report inside the job's limit
steps:
- uses: actions/checkout@<sha>
with: { persist-credentials: false } # keep the token out of .git/config
- uses: astral-sh/setup-uv@<sha>
with: { version: "<uv-version>", enable-cache: true }
- run: >-
uvx --exclude-newer <upload-time> testudos@<version>
run --parallel --no-fail-fast --strict-versions --python-preference only-managed
--junit-xml .testudos/junit --coverage --coverage-fail-under 80
- uses: actions/upload-artifact@<sha>
if: always() # failed and interrupted runs most of all
with:
name: testudos-junit
path: .testudos/junit/
include-hidden-files: true
The job owns everything around the run: services: (a database for
integration tests), a container:, steps before it (system packages, a cloud
login that yields a private index's token), a runner group.
To fan out, one job per package and Python version, so each is its own check and can be re-run on its own:
plan:
runs-on: ubuntu-latest
permissions: { contents: read }
outputs: { matrix: "${{ steps.p.outputs.matrix }}" }
steps:
- uses: actions/checkout@<sha>
with: { fetch-depth: 0, persist-credentials: false } # history for --changed-since
- uses: astral-sh/setup-uv@<sha>
with: { version: "<uv-version>" }
- id: p
# Assigned first: a command substituted into echo's argument loses its
# exit code, so a failure would pass the step and skip the test job --
# which required checks count as passing.
run: |
matrix=$(uvx --exclude-newer <upload-time> testudos@<version> run --matrix package-version --discover --strict-versions --changed-since origin/main)
echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
test:
needs: plan
if: fromJSON(needs.plan.outputs.matrix).include[0] != null # nothing to test
runs-on: ubuntu-latest
timeout-minutes: 30
permissions: { contents: read }
env:
TESTUDOS_DEADLINE: 25m
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
name: Test (${{ matrix.package }}, Python ${{ matrix.python }})
continue-on-error: ${{ matrix['allow-failure'] }}
steps:
- uses: actions/checkout@<sha>
with: { persist-credentials: false }
- uses: astral-sh/setup-uv@<sha>
with: { version: "<uv-version>", enable-cache: true }
- run: >-
uvx --exclude-newer <upload-time> testudos@<version>
run ${{ matrix.path }} --python ${{ matrix.python }} --locked
--python-preference only-managed --junit-xml .testudos/junit
- uses: actions/upload-artifact@<sha>
if: always()
with:
name: junit-${{ matrix.id }}
path: ${{ matrix.path }}/.testudos/junit/
include-hidden-files: true
A test suite is code, and on a pull request it is the pull request's code: run
it with permissions: { contents: read } and persist-credentials: false, so
it can read neither a write token nor the one actions/checkout would leave in
.git/config, and never under pull_request_target, which runs with the base
repository's secrets.
testudos run --matrix package-version (or package: one entry per package,
with python: all) prints {"include": [{"package", "path", "python", "id", "allow-failure"}, ...]} and runs nothing; every option a run takes applies, so
--include-prerelease, --exclude, --package, --python and
--allow-failure shape the matrix exactly as they would a run. An entry whose
version may fail has "allow-failure": true, for continue-on-error. GitHub
caps a matrix at 256 jobs; past that, package-version falls back to one entry
per package and says so. --format json does not change this output: for the
packages, their versions, which may fail and what --changed-since left out,
run the same command with --dry-run --format json in place of --matrix.
With one job per version, each job's coverage covers one version, so code
gated on the version reads as uncovered in every job. Run those jobs without a
threshold, keep each one's .testudos/coverage/ as an artifact (the
.coverage.* files, and the coveragerc that maps an installed package back
to its source under --no-editable; upload-artifact needs
include-hidden-files: true for them), and check the threshold once, in a job
that downloads them all into place: testudos coverage combine, then
testudos coverage report, which applies the package's own
coverage-fail-under when --fail-under is not given.
For an organisation
A baseline for testing many repositories the same way, as environment variables a shared workflow sets:
concurrency: { group: "${{ github.workflow }}-${{ github.ref }}", cancel-in-progress: true }
permissions: { contents: read }
env:
TESTUDOS_PYTHON_VERSIONS: ${{ vars.PYTHON_MATRIX }} # one org-level variable decides the matrix
TESTUDOS_STRICT_VERSIONS: "1" # never test the bundled guess
TESTUDOS_PYTHON_PREFERENCE: only-managed # the same interpreter on every runner
TESTUDOS_LOCKED: "1" # a stale uv.lock fails
TESTUDOS_REQUIRE_PINNED: "1" # no test dependency outside the lock
TESTUDOS_PARALLEL: "1"
TESTUDOS_FAIL_FAST: "0" # one run reports every failing version
TESTUDOS_JUNIT_XML: .testudos/junit # per-test annotations, and the reports
TESTUDOS_DEADLINE: 55m # inside timeout-minutes: 60
UV_INDEX_INTERNAL_USERNAME: ${{ secrets.INDEX_USER }} # credentials via env, never in URLs:
UV_INDEX_INTERNAL_PASSWORD: ${{ secrets.INDEX_TOKEN }} # setup uses them, the suite never sees them
What each line buys, and the rest of the baseline:
- One matrix.
PYTHON_MATRIXas an organisation variable, read by every repository; each project'srequires-pythonstill narrows it. Bump it once a year, when a cycle ships and one reaches end of life. - Reproducible runs. Pin actions by commit SHA, and testudos and uv by
version. Keep test dependencies in a locked group or extra (
--group test) rather than--with; a project without auv.lockresolves fresh every run, so give itexclude-newer. - Time limits.
timeout-minutesabove the worst case, and a deadline a few minutes inside it, so a run that would overrun stops itself and reports. - Early warning, not blocked merges.
--include-prerelease --allow-failure prereleasetests the next Python;include-variants = ["freethreaded"]with--allow-failure freethreadedthe free-threaded builds. - The rest of the range, nightly. A scheduled job with
--resolution highestcatches upstream breakage before consumers do; one with--resolution lowest-direct --no-sourceschecks the floors a published library promises. Add--no-editableto a published library's jobs, so the wheel is what is tested. - Monorepos.
--discover,--matrix packageand--changed-since: one job per package, only for what a change reaches. Declare the workspace ([tool.uv.workspace], or[tool.testudos.workspace]), and put shared settings in the root's[tool.testudos], which members inherit. - Routing failures. Elsewhere than GitHub, use the exit code: 1 is the answer (tests, setup or coverage failed), 3 is worth a retry, 4 goes to whoever owns the configuration.
Whatever the CI system:
- Pin testudos (
uvx testudos@<version>): every change merged here is published, so an unpinned install changes under you. A release that changes or removes anything docs/STABILITY.md calls stable says so under### Breakingin the CHANGELOG, with no deprecation period before it, so have automated bumps hold back a release with that section for a person to read. Pin its dependencies as well, by excluding anything published after the release:uvx --exclude-newer <upload-time> testudos@<version>, with any time after the release's upload to PyPI. - Pin uv (
astral-sh/setup-uvwithversion:, or[tool.uv] required-version): uv is still 0.x, and its behaviour changes between releases. --no-fail-fast, so one run reports every failing version.--lockedwhenuv.lockis committed, and test dependencies in a dependency group or extra (--group test) rather than--with, so they come from the lockfile;--require-pinnedmakes that a rule.--strict-versions, or a centralTESTUDOS_PYTHON_VERSIONS, so the matrix is never the bundled guess.--python-preference only-managedfor the same interpreter everywhere.- The default 30-minute timeout, or a
--timeoutsized for your suite; the setup phase keeps interpreter downloads out of it. --format json --output results.jsonfor a machine-readable record next to the readable log, and the exit code (below) to tell a failing suite from an outage. Every JSON document says what produced it --"generated_by": {"testudos": ..., "uv": ...}-- and so does the step summary;testudos --versionprints both for a log.
On GitHub Actions (GITHUB_ACTIONS=true, which the runner sets) testudos also
writes what GitHub reads from a job:
- each version's output inside a
::group::, so the log folds to one line per version, and inside a::stop-commands::block with a random token, so a line the suite prints that starts with::-- a fake::error, an::add-mask::-- is shown as text rather than obeyed; - with
--junit-xml, an::errorannotation for each failing test, on its line in the pull request's diff and naming every version it fails on (tests.test_x::test_parse -- fails on 3.12, 3.13): a test that fails everywhere is one annotation, not one per version; - an
::errorannotation for each failed version the tests do not account for -- a setup failure, a timeout, a suite that is not pytest -- and for a missed coverage threshold, and a::warningfor each warning (a fallback matrix, an unknown config key). These show on the run's summary page and the pull request's checks. GitHub shows at most ten error annotations per step, so the tests failing on the most versions come first and the last annotation says how many more there are; - a results table appended to
$GITHUB_STEP_SUMMARY, with a grid of the failing tests by Python version (❌ failed, ✅ passed, – not run), cut to fit GitHub's 1 MiB per step -- the tests failing most widely kept, and a line saying how many more there are.
--ci-format github turns this on anywhere, --ci-format none turns it off.
testudos run --junit-xml .testudos/junit # a JUnit XML report per version
For pytest, each version writes .testudos/junit/<version>.xml itself, with a
suite named python-<version> -- <package>/python-<version> when the run has
more than one package, so a report viewer reading every file keeps them apart
-- which test-report actions and GitLab or Jenkins read. A version pytest did
not write one for -- its setup failed, it timed out, fail-fast skipped it, or
the suite is not pytest -- gets a one-test-case report from testudos, so no
version is missing. A relative directory is inside each package; an absolute
one gets a subdirectory per package, named after the package (two packages can
share a directory name, never a package name). The paths are in the JSON as
"junit_xml".
In a monorepo, test only what a change reaches:
testudos run --discover --changed-since origin/main
A package is tested when a file inside it changed since the branch left
origin/main (committed, uncommitted or untracked), or when it depends on one
that did -- through [project].dependencies, optional dependencies, dependency
groups, [tool.uv.sources], Poetry's dependency tables and groups, or the
requirement files setuptools reads for dynamic dependencies (-r and -e <path> included) -- directly or through other packages. A package whose
dependencies are dynamic with nothing readable behind them (a setup.py
computing them) is assumed to depend on every package. A change
outside every package (CI configuration, say) tests all of them, and so does a
change a package owns that none of the packages nested inside it does: when the
repository root is itself a package, its uv.lock or conftest.py reaches
every member. The rest are listed as not affected, and as "not_affected" in
JSON, with the reason each tested package was picked in "affected_by" (and in
--verbose output); with nothing affected the run exits 0.
Rules in the repository root's pyproject.toml refine that, as globs written
the way .gitignore writes them:
[tool.testudos.changed-since]
ignore = ["*.md", "docs/**"] # these changes affect no package
global = ["libs/shared/fixtures/**"] # these affect every package
It needs the ref's history: actions/checkout with fetch-depth: 0. Compare a
pull request against its base (origin/${{ github.base_ref }}) and a merge
queue entry against github.event.merge_group.base_sha; leave changed-since
empty on pushes to the default branch and on scheduled runs, so the default
branch is always tested in full. A ref that cannot be compared against -- the
null commit GitHub reports as github.event.before for a push that creates a
branch, a commit a force-push left behind, a ref a shallow clone lacks -- tests
every package, with a warning saying why, rather than failing the run.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Every version passed, and coverage met its threshold |
| 1 | Tests failed, an environment could not be set up (a stale lock under --locked, requirements that do not resolve, a build that fails), or coverage missed --coverage-fail-under |
| 2 | Interrupted: Ctrl-C, SIGTERM or SIGHUP |
| 3 | testudos could not run: uv missing or older than 0.4.27, no version data under --strict-versions, an unwritable --output, or testudos itself crashed |
| 4 | The command line or a configuration is wrong: an unknown option or value, an invalid [tool.testudos], a requires-python nothing satisfies, a missing coverage source |
CI can retry on 3, send 4 to whoever owns the configuration, and treat 1 as the result. A setup failure is a 1 rather than a 3 because retrying it almost never helps: the lockfile, the requirements or the build is what needs fixing.
Environment variables
Most run options can be set from the environment, which is how shared CI
templates are usually configured; a flag on the command line still wins.
testudos run --help lists them: TESTUDOS_TIMEOUT, TESTUDOS_PARALLEL,
TESTUDOS_JOBS, TESTUDOS_MAX_CONCURRENCY, TESTUDOS_FAIL_FAST,
TESTUDOS_COVERAGE, TESTUDOS_COVERAGE_REPORT, TESTUDOS_COVERAGE_FAIL_UNDER,
TESTUDOS_WITH, TESTUDOS_GROUP, TESTUDOS_EXTRA, TESTUDOS_LOCKED,
TESTUDOS_FROZEN, TESTUDOS_EXCLUDE_NEWER, TESTUDOS_PYTHON_PREFERENCE,
TESTUDOS_SETUP, TESTUDOS_SETUP_TIMEOUT, TESTUDOS_DEFAULT_INDEX,
TESTUDOS_INDEX, TESTUDOS_TEST_ARGS, TESTUDOS_ALLOW_FAILURE, TESTUDOS_COVERAGE_VERSION, TESTUDOS_DEADLINE, TESTUDOS_FORMAT, TESTUDOS_OUTPUT, TESTUDOS_REQUIRE_PINNED,
TESTUDOS_RESOLUTION, TESTUDOS_NO_SOURCES, TESTUDOS_EDITABLE.
TESTUDOS_UNTESTED_FLOOR (warn, ignore, error) is read the same way, below a
project's own untested-floor.
Repeatable options take a space-separated list (TESTUDOS_GROUP="test lint").
TESTUDOS_OFFLINE, TESTUDOS_STRICT_VERSIONS, TESTUDOS_PYTHON_VERSIONS,
TESTUDOS_ENDOFLIFE_URL and TESTUDOS_TRACEBACK are described with their
features above.
Proxies and certificates
testudos makes one HTTP request, to endoflife.date, with Python's standard
library: it honours HTTPS_PROXY and NO_PROXY and verifies TLS against the
operating system's trust store, so a TLS-inspecting corporate proxy whose
certificate the machine trusts works without extra configuration. uv has its
own settings for the package index (UV_NATIVE_TLS / UV_SYSTEM_CERTS).
Configuration
Everything above can be set in [tool.testudos]:
[project]
requires-python = ">=3.11"
[tool.testudos]
# Version selection
python-versions = ["3.11", "3.12", "3.13"] # override auto-detection
include-prerelease = false # include unreleased cycles
include-variants = ["freethreaded"] # add 3.13t, 3.14t, ... to auto-detection
allow-failure = ["prerelease", "freethreaded"] # report these, but do not fail on them
offline = false # never call endoflife.date
strict-versions = false # fail rather than test the bundled list
python-preference = "only-managed" # uv-managed interpreters only
# What to run
test-command = "pytest"
test-args = ["-v", "--tb=short"]
test-groups = ["test"] # [dependency-groups] to install
test-extras = ["test"] # extras to install
test-dependencies = ["pytest-asyncio"] # extra packages per isolated env
timeout = 300 # seconds per version (0: none)
setup = true # install each environment first
setup-timeout = 900 # seconds per version's setup
# The lockfile
locked = true # fail when uv.lock is stale
frozen = false # use uv.lock without checking it
require-pinned = false # refuse unpinned --with requirements
untested-floor = "warn" # requires-python admitting EOL versions: warn, ignore, error
resolution = "locked" # or highest / lowest-direct, resolved outside uv.lock
no-sources = false # ignore [tool.uv.sources]
editable = true # false: test the built wheel
exclude-newer = "2026-10-01" # ignore releases after this date
# How to run it
junit-xml = ".testudos/junit" # a JUnit XML report per version
parallel = false
max-jobs = 4
max-concurrency = 8 # uv processes at once (default: CPUs)
# Coverage
coverage = false
coverage-combine = true
coverage-report = ["term"]
coverage-fail-under = 80
coverage-source = ["src"]
# Package indexes
default-index = "https://pypi.internal.example.com/simple"
index = ["https://extra.example.com/simple"]
find-links = ["/path/to/wheels"]
no-index = false
index-strategy = "first-index"
# --changed-since rules (read from the repository root's pyproject.toml)
# The workspace's packages, for a monorepo uv does not manage
[tool.testudos.workspace]
members = ["libs/*", "apps/*"]
exclude = ["libs/legacy"]
[tool.testudos.changed-since]
ignore = ["*.md", "docs/**"] # changes that affect no package
global = [] # changes that affect every package
CLI options win over configuration. Values are validated on load: a wrong type or an out-of-range value is an error, and an unrecognised key is a warning.
test-command is a command line, not just a program name, so a runner that
needs arguments to name itself fits in one setting:
test-command = "pytest"
test-command = "python -m unittest discover"
test-command = "./run-tests"
It is split the way a shell would split it (quoted words stay together), but
no shell runs it: the pieces become an argv list. Only the first word is a
package name, so it is what gets installed into each isolated environment —
a suite run as python -m pytest should name pytest in test-dependencies,
since the interpreter is what testudos sees. A first word containing a /,
like ./run-tests, is a file in your project rather than a package, so
nothing is installed for it.
test-args are passed to that command exactly as written, with no filtering,
so metacharacters need no escaping and no workaround:
test-args = ["--ignore-glob=*$py.class", "-W", "error:.*deprecated$:UserWarning"]
By the same token nothing in them is expanded either — --rootdir=$PWD
arrives as the literal string $PWD, not as your working directory.
Documentation
- Architecture — how the pieces fit together, and the decisions behind them
- Stability — what a version bump can and cannot change, and how breaking changes are announced
Development
git clone https://github.com/martinristovski/testudos.git
cd testudos
uv sync --extra dev
just ci # lint, type-check, test, network tests, dogfood, version check
just test # pytest with coverage
just fix # auto-fix lint and formatting
just dogfood # run testudos on itself
The default test suite is hermetic: the endoflife.date API is stubbed and the
version cache is redirected per test, so no test touches the network or your
real ~/.cache/testudos.
A separate set of tests does reach the network: most exercise the real uv
boundary, and so resolve packages from PyPI, and one checks the canned
endoflife.date response against the live API. They are marked network and deselected by default; run
them with pytest -m network. CI runs them as their own step, so an index or
API outage fails a step that names the cause.
License
MIT
Metadata
Release files for testudos 0.70.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| testudos-0.70.0.tar.gz | 468.6 kB | With provenance | Details for testudos-0.70.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| testudos-0.70.0-py3-none-any.whl | Python 3 | none | any | Details for testudos-0.70.0-py3-none-any.whl |
Total release size: 648.6 kB
Release files / testudos-0.70.0.tar.gz
| Download URL | testudos-0.70.0.tar.gz |
|---|---|
| Size | 468.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d3ef7c37cfdc4354ae58af73674a77fc1df66f238a04ce0dfd09b488830f4b78
|
|
BLAKE2b-256 checksum How to use checksums |
457f52a0f3871da4ba2358a73852f570a1bfc35ea3ff132c362f63ef164e85d8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logRelease files / testudos-0.70.0-py3-none-any.whl
| Download URL | testudos-0.70.0-py3-none-any.whl |
|---|---|
| Size | 180.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
07dd81f8a71367f782005101b1f65a3dd4ccb072d6bb558572d5851551bb200f
|
|
BLAKE2b-256 checksum How to use checksums |
0b920df299ad4a79e0ea054802617dc237c194408b7729fa6cbf8994d7bb5ea7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logSecurity for testudos 0.70.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for testudos-0.70.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.