skippy-cov
Repository: mgaitan/skippy-cov. Supports Python 3.12–3.15. Development uses uv, Ruff, ty and optional prek hooks; see Contributing.
Overview
skippy-cov is a tool that optimizes your Python testing workflow by filtering tests based on code changes. It takes a diff file from git and a .coverage database (generated by coverage.py) to identify and run only the tests relevant to the modified code. This significantly reduces testing time and resources.
Usage
To use this, you first need to generate the following required files:
- Run Coverage Analysis: Generate a
.coveragedatabase usingcoverage.pyviapytest-covwith test contexts
pytest --cov=myproj --cov-context=test myproj/tests/
Then change your code and run
pytest --skippy-cov
In this case, skippy-cov will analyze the changes and run only the tests that are relevant to the modified code.
You can also use skippy-cov as a CLI on its own.
skippy-cov
that will list the tests selected.
Configuration
The inputs for skippy-cov are the coverage file (default: .coverage) and the diff information. By default, the diff is generated in the same way as a pull request (PR) on GitHub: only the changes introduced by your branch compared to the target branch are considered. This is done using the git diff <target>...HEAD (triple-dot) syntax.
You can specify the coverage file and a branch or ref to diff against (e.g., the main branch):
skippy-cov --diff main --coverage-file .coverage
or equivalently
pytest --skippy-cov --skippy-cov-diff main --skippy-cov-coverage-file .coverage
You can also provide a diff file:
git diff > changes.diff
skippy-cov --diff changes.diff --coverage-file .coverage
Single refs always use <ref>...HEAD; explicit .. and ... ranges are preserved.
Git refs compare committed changes. Use a patch file as above to include working-tree
changes. Default-branch detection uses local refs and does not contact the remote.
Without origin/HEAD, it checks local main/master branches, then fetched
origin/main/origin/master refs. Fetch the required refs before selecting tests
in a shallow CI checkout; the comparison still needs a common ancestor.
CLI output is deterministic. --format lines emits one target per line;
--format json maps each test module to its selected node IDs, where an empty list
means the entire module. Entries for the same module are merged. The default
format quotes paths for copying into a shell command; prefer JSON for automation.
--relative-to accepts files and directories without mutating the selection;
--strip-prefix requires a single scope. Pytest's legacy
--skippy-cov-strip-prefix option is deprecated because the plugin preserves the
caller's resolved scopes and exact node IDs.
If you omit the --diff argument, it will default to the main branch as determined by your git remote (usually "main" or "master").
Advanced:
You may pass any valid git diff refspec to --diff, including triple-dot syntax (e.g., main...HEAD or origin/master...feature-branch). If you provide a triple-dot ref, it will be used as-is, giving you full control over the comparison range.
See skippy-cov --help for more information.
Pytest selection reports
With --skippy-cov, the terminal summary reports the decision: applied,
fallback (the original selection was retained), empty, or incomplete
(distributed decisions could not be established reliably). It includes a
reason and test counts when available. Module mode narrows paths before
collection, so the number of tests excluded is unknown; it does not collect the
original suite just to count them. Test mode counts items after pytest's scope,
-k and -m filters; deselected counts include only skippy-cov's removals.
Counts describe selection, not test outcomes.
Optionally write the same decision for automation:
pytest --skippy-cov --skippy-cov-report=selection.json
The UTF-8 JSON schema (version 1) has these fixed fields:
| Field | Type | Meaning |
|---|---|---|
schema_version |
integer | Currently 1 |
status |
string | applied, fallback, or empty |
reason_code |
string | Stable code described below |
reason |
string | Human-readable detail; not a machine interface |
granularity |
string | module or test |
selected |
integer or null | Retained collected items, or 0 for an early empty decision |
deselected |
integer or null | Items removed by skippy-cov; unknown in module narrowing |
baseline |
object | coverage_file (string), source_root and base_ref (string or null), as supplied |
Reason codes are coverage_selection (applied), no_affected_tests (empty),
selection_error (unsafe/unavailable inputs or impact), outside_checkout
(test items outside the checkout), and selection_incomplete (interruption
before a decision completed). Baseline fields are input metadata, not verified
coverage provenance; no revision is inferred from the database. Unavailable
counts are null, including before collection completes.
With pytest-xdist, only the controller writes JSON and the terminal summary.
Workers transfer their collection decisions to the controller. Matching worker
reports are counted once, not summed: collection is replicated on every worker.
Missing or disagreeing reports, or worker crashes, produce status incomplete
with reason code worker_reports_incomplete and null counts. This is a reporting
uncertainty, not a claim that the original suite ran or a change to selection
or xdist's handling of failures. Module mode's early empty exit is reported by
the controller before workers start.
Reports are written at session finish, including failing test runs, or before
module mode's early empty-selection exit (status 5). The destination's parent
must already exist; existing files are overwritten. Write failures emit a
PytestWarning rather than replacing pytest's outcome (normal pytest warning
filters still apply). Without --skippy-cov, the report option has no effect:
no file or summary is emitted. These additions do not change the standalone
CLI or GitHub Action interfaces, selection scopes, or exit codes.
Coverage baseline
Run skippy-cov from any directory inside a Git checkout. Diff paths and source inspection use the actual Git checkout root (not pytest's configuration root). Without Git, the current directory is used.
User-supplied coverage-file paths (including the default .coverage), patch-file
paths, --source-root, and explicit test scopes remain relative to the invocation
directory. CLI output is also invocation-relative; --strip-prefix instead strips
the requested --relative-to scope.
By default, relative coverage filenames and pytest context paths are interpreted
relative to the checkout root, regardless of the invocation directory. For a
relative_files = true baseline recorded from a checkout subdirectory, explicitly
set --source-root packages/api (pytest: --skippy-cov-source-root packages/api)
when invoking from the checkout root, or --source-root . when invoking from
packages/api. This rebases both source records and test contexts into that
checkout subdirectory. The recording setup must use the same base for relative
source and context paths; differing bases are unsupported.
For absolute relocated artifacts, a source root outside the current checkout is the original checkout root, whose prefix is replaced by the current checkout root. Absolute records under an in-checkout source root retain that subdirectory prefix too. Unmappable absolute records are rejected.
Relative coverage contains no recording-directory metadata. Neither the current invocation directory nor basename matches establish its origin. A missing exact match with a relative suffix in the baseline raises an actionable recording-root error (the plugin falls back to the original selection), never a guessed match. This cannot detect every mismatch: colliding paths may be valid under either origin. You must provide the recording directory explicitly for subdirectory baselines, even if a same-named file exists at the checkout root.
Pytest stays in the invocation directory: configuration discovery, default
collection, -k, -m, and exact node IDs retain their normal pytest semantics.
Selection never expands the caller's scope. The pytest plugin warns and keeps
the original test selection when baseline validation fails.
The baseline must be a readable coverage database with pytest test contexts,
generated using --cov-context=test. Missing, empty, corrupt or context-free
data is an error; the CLI exits with status 2 rather than printing an apparently
valid empty selection.
Validation does not establish baseline completeness or revision provenance. The caller must use data from a successful complete test run, retain its commit SHA, and compare against that revision, including intervening changes to the base branch. A readable database alone does not guarantee a trustworthy selection.
Prefer portable coverage paths when collecting data in another checkout or Docker:
[tool.coverage.run]
relative_files = true
core = "ctrace"
If a baseline already contains absolute paths from another checkout, provide its original root explicitly:
skippy-cov --coverage-file baseline/.coverage --source-root /fierro
pytest --skippy-cov --skippy-cov-coverage-file baseline/.coverage --skippy-cov-source-root /fierro
Paths are matched relative to the checkout root, never guessed from basenames.
Every recorded source path must map to that root. Mixed baselines from multiple
absolute checkout roots are rejected rather than silently losing test consumers;
combine them with coverage's path aliases or collect relative paths first.
On Python 3.14+, coverage defaults to
sysmon, which does not support dynamic test contexts; collect the baseline with
core = "ctrace" as shown above.
Conservative selection and fallback
By default, selection runs whole test modules. Pytest discovers unittest classes, inherited methods, custom function names and parameterizations; skippy-cov does not rebuild their node IDs with a separate AST visitor.
Individual test selection (opt-in)
The default granularity is module. Individual selection is an opt-in heuristic:
caches and state shared between tests can hide consumers from coverage. Keep
module selection as the default and run the complete suite before merging.
To select individual tests, use:
pytest --skippy-cov --skippy-cov-diff=BASELINE_SHA..HEAD --skippy-cov-granularity=test
skippy-cov --diff BASELINE_SHA..HEAD --granularity test --format json
The coverage database must have been collected at BASELINE_SHA, with
--cov-context=test. Its revision cannot be inferred from a coverage database
alone; providing coverage from a different commit can produce an incorrect
selection. A single Git ref uses its merge base with HEAD, while an explicit
BASELINE_SHA..HEAD range uses that exact baseline. The new revision must be
HEAD. With a patch file, provide --base-ref BASELINE_SHA to the CLI or
--skippy-cov-base-ref=BASELINE_SHA to pytest so the old source is available.
This mode parses function, async-function, method, nested-function, and class
ranges from the old source without executing it. A function-body change selects
the tests recorded anywhere in that function's baseline runtime body, including
consumers in other test files. It uses old line numbers for coverage, so shifted
definitions and insertions into previously uncovered branches remain eligible.
Unrelated sibling functions and methods do not share a selection.
Decorated functions and methods inside decorated scopes require the original
selection: a decorator such as functools.lru_cache can return a value without
executing the recorded function body for every consumer.
Every new test is included independently of coverage and source impact.
Changed test modules run in full, including new functions, inherited methods,
and new parameter cases. Removing lines without known runtime coverage, including
parameter declarations, requires the original selection. The plugin keeps pytest's
original collection and then deselects unaffected tests, respecting requested paths, exact node IDs, -k,
and -m. For an affected parameterized test it runs all currently collected
cases within that original scope. The CLI resolves current IDs through an
isolated pytest collection; pytest must be installed for this mode. It keeps
stdout suitable for the selected output format and protects the baseline
coverage database during collection. Test paths/discovery configuration apply;
the CLI's collection clears execution addopts and PYTEST_ADDOPTS. Apply
execution filters to the eventual pytest invocation, or use the plugin to keep
the full original pytest argument semantics.
For independent suites, repeat --collect-group to collect each group in a
separate Python process. All groups are combined before impact selection. Pass
collection options explicitly, including options normally set in addopts:
skippy-cov --diff BASELINE_SHA..HEAD --granularity test --format json \
--collect-group tests/client tests/shared --collect-group tests/server \
--pytest-arg=--import-mode=importlib
Each group accepts one or more paths; use the resulting node IDs with the
original suite commands. --relative-to still filters the final output, so omit
it when all groups must be represented. Pytest arguments apply to every group;
configuration addopts and PYTEST_ADDOPTS remain cleared. Pass only arguments
needed for collection, without execution or coverage options. A failed group or
unsafe impact exits nonzero rather than returning a partial selection; an empty
JSON object is a successful empty selection. Collection imports test modules,
but does not run tests or fixtures, and uses a temporary coverage database.
New standalone functions, signature/decorator changes, class attributes/bases, module declarations, ambiguous scope matches, unknown/import-time coverage, and shared setup/teardown require the original selection. Recorded file consumers cannot establish every import-time dependency, such as a constant imported by another test module. The plugin preserves the original selection with a diagnostic; the CLI reports that it cannot select safely. New source files and non-Python changes retain that full-selection fallback. The test suite compares individual, module, and full runs with real coverage, including a shared-session-fixture regression where narrowing could hide a failing second consumer. Historical coverage remains an impact heuristic.
Module selection rules
Changes to existing source function bodies with recorded execution under a
test's run phase select the consuming test modules. If a source diff also
changes imports, declarations, or otherwise has unknown impact,
skippy-cov preserves the original selection. Shared fixtures may
execute only for the first test that requests them, so those contexts cannot
identify every test that reuses their results. A file-level fallback involving
setup/teardown therefore requires the original pytest selection, as does a file
with no recorded consumers. This does not prevent selection for validated
function-body changes covered only during run in the same source file.
Changes to test modules select the whole module when edits stay within test
functions, including newly added tests and pytest.mark.parametrize cases.
Removing lines with unknown or import-time coverage requires the original
selection; comments and blank-looking text inside string literals are code.
Imports, other decorators, fixtures, and recorded setup/teardown execution
still require the original selection because they can affect shared test setup.
Conftest files, non-Python inputs, new/deleted/renamed files, and unknown
coverage also require the original selection. Recorded consumers of inherited
tests are included alongside the edited module. Separate edit blocks are
validated independently, even when Git groups them into one hunk.
An empty diff or no affected tests inside the caller's scope exits with pytest
status 5, not a successful test run.
The pytest plugin warns and keeps the original arguments when the baseline or
impact is unknown. The CLI exits with status 2 for those cases: callers must run
their full test command instead of treating an error as an empty selection.
Explicit node IDs and absolute paths passed to pytest retain their scope.
An explicit --rootdir does not change the base used for selected module paths.
Unresolvable Git refs also warn and preserve the original pytest selection.
Historical coverage is a heuristic, not a proof of all dependencies. Reuse data only from a successful complete run, keep its commit SHA alongside the artifact, and compare against that commit, including intervening changes to the base branch. Dynamic imports, cross-test state and shared fixtures still warrant periodic full runs and a complete suite before merging.
GitHub Actions
The composite action at .github/actions/select finds the newest accessible
coverage cache from a successful full workflow on the target branch whose commit
is an ancestor of both the checkout and target ref. It verifies the cached SHA,
then returns baseline, selected, targets (JSON) and reason. A missing,
invalid or unsafe baseline returns selected: false; the caller chooses its
full workflow. An empty JSON object is a successful empty selection.
GitHub metadata is read directly through the REST API using Python's standard
library; GitHub CLI (gh) is not required. The action uses the job token and
GITHUB_API_URL (including GitHub Enterprise), paginates cache results, and
falls back safely when the API cannot provide a valid baseline.
After all full-suite jobs have succeeded, combine their coverage databases and save the baseline directory in the full workflow:
- run: |
mkdir -p .skippy-baseline
cp .coverage .skippy-baseline/.coverage
printf '%s\n' "$GITHUB_SHA" > .skippy-baseline/sha.txt
- uses: actions/cache/save@v4
with:
path: .skippy-baseline
key: skippy-baseline-v1-${{ github.run_id }}-${{ github.sha }}
Save only on pushes to the target branch; the action also verifies that the
whole workflow ultimately succeeded. Configure the PR job with actions: read
and contents: read, a checkout with an origin remote,
and the project environment with pytest and a version of skippy-cov containing
this action's collection options. For example:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- id: skippy
uses: mgaitan/skippy-cov/.github/actions/select@<release-tag>
with:
base-ref: ${{ github.event.pull_request.base.sha }}
base-branch: ${{ github.event.pull_request.base.ref }}
workflow: .github/workflows/ci-full.yml
cache-prefix: skippy-baseline-v1
history-depth: 100
collection-groups: '[["tests/client"], ["tests/server"]]'
pytest-args: '["--import-mode=importlib"]'
# source-root: /original/container/checkout
- if: steps.skippy.outputs.selected != 'true'
run: echo "Full suite required: $REASON"
env:
REASON: ${{ steps.skippy.outputs.reason }}
For shallow checkouts, the action performs one git fetch --no-tags --depth=N
from origin for the checked-out HEAD and target commit, where N is
history-depth (default 100, a positive integer). Prefer the PR's exact base
SHA as above; a target ref must otherwise be available locally or fetchable from
origin. The fetched target is stored in refs/skippy-cov/base, without changing
the checkout or local branches. Complete checkouts with an available target do
not fetch. This bounds history depth per tip, not total transfer bytes (merge
history and repository contents can still be large).
The action never automatically deepens again or downloads the full history.
It selects a cache only when Git proves its revision is an ancestor of both
HEAD and the target in the available history. An older baseline beyond the
shallow boundary, unrelated history, or a failed fetch yields selected: false
and the full-suite fallback, never an assumed ancestry. Increase history-depth
if baselines are commonly older; fetch-depth: 0 remains an explicit option
when full history is preferred.
Use the exact same baseline-path when saving and restoring: GitHub cache
versions include the path. The cache prefix must identify a compatible test
environment and complete suite. Capture per-test coverage with
--cov-context=test for individual selection. Collection runs in the action's
job environment, so install the project's test dependencies there or run the
job in the appropriate container. Test execution, runner setup, service setup,
and decisions about empty selections remain with the caller.
Contributing
See CONTRIBUTING.md for information on how to contribute to the project.
License
See LICENSE for licensing information.
Metadata
Release files for skippy-cov 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| skippy_cov-0.5.1.tar.gz | 71.1 kB | With provenance | Details for skippy_cov-0.5.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| skippy_cov-0.5.1-py3-none-any.whl | Python 3 | none | any | Details for skippy_cov-0.5.1-py3-none-any.whl |
Total release size: 97.1 kB
Release files / skippy_cov-0.5.1.tar.gz
| Download URL | skippy_cov-0.5.1.tar.gz |
|---|---|
| Size | 71.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
df8e57f71fafd9533b7d191c87d0af70d8b796bbe7feac7481c5ea8363ddab45
|
|
BLAKE2b-256 checksum How to use checksums |
188b9fb185aed9b40d5c780511b6949ec42cdc817e81783c1f05e5f2decc0ba8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logRelease files / skippy_cov-0.5.1-py3-none-any.whl
| Download URL | skippy_cov-0.5.1-py3-none-any.whl |
|---|---|
| Size | 26.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2d080a2f4ddb8f4727b2b0f9a57b500d7919a589df7b3a0a21a23f6de99e7041
|
|
BLAKE2b-256 checksum How to use checksums |
6cd53230eb20e8142760c0da7c62595f1f6ae155bff63b3e80fdee801a3ca404
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logSecurity for skippy-cov 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for skippy_cov-0.5.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.