skippy-cov

Repository: mgaitan/skippy-cov. Supports Python 3.12–3.15. Development uses uv, Ruff, ty and optional prek hooks; see Contributing.

Overview

skippy-cov is a tool that optimizes your Python testing workflow by filtering tests based on code changes. It takes a diff file from git and a .coverage database (generated by coverage.py) to identify and run only the tests relevant to the modified code. This significantly reduces testing time and resources.

Usage

To use this, you first need to generate the following required files:

• Run Coverage Analysis: Generate a .coverage database using coverage.py via pytest-cov with test contexts

    pytest --cov=myproj --cov-context=test myproj/tests/

Then change your code and run

pytest --skippy-cov

In this case, skippy-cov will analyze the changes and run only the tests that are relevant to the modified code.

You can also use skippy-cov as a CLI on its own.

skippy-cov

that will list the tests selected.

Configuration

The inputs for skippy-cov are the coverage file (default: .coverage) and the diff information. By default, the diff is generated in the same way as a pull request (PR) on GitHub: only the changes introduced by your branch compared to the target branch are considered. This is done using the git diff <target>...HEAD (triple-dot) syntax.

You can specify the coverage file and a branch or ref to diff against (e.g., the main branch):

skippy-cov --diff main --coverage-file .coverage

or equivalently

pytest --skippy-cov --skippy-cov-diff main --skippy-cov-coverage-file .coverage

You can also provide a diff file:

git diff > changes.diff
skippy-cov --diff changes.diff --coverage-file .coverage

Single refs always use <ref>...HEAD; explicit .. and ... ranges are preserved. Git refs compare committed changes. Use a patch file as above to include working-tree changes. Default-branch detection uses local refs and does not contact the remote. Without origin/HEAD, it checks local main/master branches, then fetched origin/main/origin/master refs. Fetch the required refs before selecting tests in a shallow CI checkout; the comparison still needs a common ancestor.

CLI output is deterministic. --format lines emits one target per line; --format json maps each test module to its selected node IDs, where an empty list means the entire module. Entries for the same module are merged. The default format quotes paths for copying into a shell command; prefer JSON for automation. --relative-to accepts files and directories without mutating the selection; --strip-prefix requires a single scope. Pytest's legacy --skippy-cov-strip-prefix option is deprecated because the plugin preserves the caller's resolved scopes and exact node IDs.

If you omit the --diff argument, it will default to the main branch as determined by your git remote (usually "main" or "master").

Advanced: You may pass any valid git diff refspec to --diff, including triple-dot syntax (e.g., main...HEAD or origin/master...feature-branch). If you provide a triple-dot ref, it will be used as-is, giving you full control over the comparison range.

See skippy-cov --help for more information.

Coverage baseline

Run skippy-cov from the repository root. Invocation from a Git subdirectory is rejected with an explicit diagnostic. The pytest plugin warns and keeps the original test selection when baseline validation fails.

The baseline must be a readable coverage database with pytest test contexts, generated using --cov-context=test. Missing, empty, corrupt or context-free data is an error; the CLI exits with status 2 rather than printing an apparently valid empty selection.

Validation does not establish baseline completeness or revision provenance. The caller must use data from a successful complete test run, retain its commit SHA, and compare against that revision, including intervening changes to the base branch. A readable database alone does not guarantee a trustworthy selection.

Prefer portable coverage paths when collecting data in another checkout or Docker:

[tool.coverage.run]
relative_files = true
core = "ctrace"

If a baseline already contains absolute paths from another checkout, provide its original root explicitly:

skippy-cov --coverage-file baseline/.coverage --source-root /fierro
pytest --skippy-cov --skippy-cov-coverage-file baseline/.coverage --skippy-cov-source-root /fierro

Paths are matched relative to the checkout root, never guessed from basenames. Every recorded source path must map to that root. Mixed baselines from multiple absolute checkout roots are rejected rather than silently losing test consumers; combine them with coverage's path aliases or collect relative paths first. On Python 3.14+, coverage defaults to sysmon, which does not support dynamic test contexts; collect the baseline with core = "ctrace" as shown above.

Conservative selection and fallback

Selection runs whole test modules. Pytest discovers unittest classes, inherited methods, custom function names and parameterizations; skippy-cov does not rebuild their node IDs with a separate AST visitor.

Only changes to existing function bodies with recorded execution under a test's run phase are eligible. Import-time execution, setup/teardown coverage, declarations, conftest files, non-Python inputs, new/deleted/renamed files and unknown coverage require the original pytest selection. Changes to test modules are eligible only within existing undecorated test bodies; decorated functions fall back because their decorators can register shared fixtures. Recorded setup/teardown execution is rejected even inside test modules, and recorded consumers of inherited tests are included alongside the edited module. Separate edit blocks are validated independently, even when Git groups them into one hunk. An empty diff or no affected tests inside the caller's scope exits with pytest status 5, not a successful test run.

The pytest plugin warns and keeps the original arguments when the baseline or impact is unknown. The CLI exits with status 2 for those cases: callers must run their full test command instead of treating an error as an empty selection. Explicit node IDs and absolute paths passed to pytest retain their scope. An explicit --rootdir does not change the base used for selected module paths. Unresolvable Git refs also warn and preserve the original pytest selection.

Historical coverage is a heuristic, not a proof of all dependencies. Reuse data only from a successful complete run, keep its commit SHA alongside the artifact, and compare against that commit, including intervening changes to the base branch. Dynamic imports, cross-test state and shared fixtures still warrant periodic full runs and a complete suite before merging.

Contributing

See CONTRIBUTING.md for information on how to contribute to the project.

License

See LICENSE for licensing information.