Skip to main content

skippy-cov

Repository: mgaitan/skippy-cov. Supports Python 3.12–3.15. Development uses uv, Ruff, ty and optional prek hooks; see Contributing.

Overview

skippy-cov is a tool that optimizes your Python testing workflow by filtering tests based on code changes. It takes a diff file from git and a .coverage database (generated by coverage.py) to identify and run only the tests relevant to the modified code. This significantly reduces testing time and resources.

Usage

To use this, you first need to generate the following required files:

  • Run Coverage Analysis: Generate a .coverage database using coverage.py via pytest-cov with test contexts
    pytest --cov=myproj --cov-context=test myproj/tests/

Then change your code and run

pytest --skippy-cov

In this case, skippy-cov will analyze the changes and run only the tests that are relevant to the modified code.

You can also use skippy-cov as a CLI on its own.

skippy-cov

that will list the tests selected.

Configuration

The inputs for skippy-cov are the coverage file (default: .coverage) and the diff information. By default, the diff is generated in the same way as a pull request (PR) on GitHub: only the changes introduced by your branch compared to the target branch are considered. This is done using the git diff <target>...HEAD (triple-dot) syntax.

You can specify the coverage file and a branch or ref to diff against (e.g., the main branch):

skippy-cov --diff main --coverage-file .coverage

or equivalently

pytest --skippy-cov --skippy-cov-diff main --skippy-cov-coverage-file .coverage

You can also provide a diff file:

git diff > changes.diff
skippy-cov --diff changes.diff --coverage-file .coverage

Single refs always use <ref>...HEAD; explicit .. and ... ranges are preserved. Git refs compare committed changes. Use a patch file as above to include working-tree changes. Default-branch detection uses local refs and does not contact the remote. Without origin/HEAD, it checks local main/master branches, then fetched origin/main/origin/master refs. Fetch the required refs before selecting tests in a shallow CI checkout; the comparison still needs a common ancestor.

CLI output is deterministic. --format lines emits one target per line; --format json maps each test module to its selected node IDs, where an empty list means the entire module. Entries for the same module are merged. The default format quotes paths for copying into a shell command; prefer JSON for automation. --relative-to accepts files and directories without mutating the selection; --strip-prefix requires a single scope. Pytest's legacy --skippy-cov-strip-prefix option is deprecated because the plugin preserves the caller's resolved scopes and exact node IDs.

If you omit the --diff argument, it will default to the main branch as determined by your git remote (usually "main" or "master").

Advanced: You may pass any valid git diff refspec to --diff, including triple-dot syntax (e.g., main...HEAD or origin/master...feature-branch). If you provide a triple-dot ref, it will be used as-is, giving you full control over the comparison range.

See skippy-cov --help for more information.

Pytest selection reports

With --skippy-cov, the terminal summary reports the decision: applied, fallback (the original selection was retained), empty, or incomplete (distributed decisions could not be established reliably). It includes a reason and test counts when available. Module mode narrows paths before collection, so the number of tests excluded is unknown; it does not collect the original suite just to count them. Test mode counts items after pytest's scope, -k and -m filters; deselected counts include only skippy-cov's removals. Counts describe selection, not test outcomes.

Optionally write the same decision for automation:

pytest --skippy-cov --skippy-cov-report=selection.json

The UTF-8 JSON schema (version 1) has these fixed fields:

Field Type Meaning
schema_version integer Currently 1
status string applied, fallback, or empty
reason_code string Stable code described below
reason string Human-readable detail; not a machine interface
granularity string module or test
selected integer or null Retained collected items, or 0 for an early empty decision
deselected integer or null Items removed by skippy-cov; unknown in module narrowing
baseline object coverage_file (string), source_root and base_ref (string or null), as supplied

Reason codes are coverage_selection (applied), no_affected_tests (empty), selection_error (unsafe/unavailable inputs or impact), outside_checkout (test items outside the checkout), and selection_incomplete (interruption before a decision completed). Baseline fields are input metadata, not verified coverage provenance; no revision is inferred from the database. Unavailable counts are null, including before collection completes.

With pytest-xdist, only the controller writes JSON and the terminal summary. Workers transfer their collection decisions to the controller. Matching worker reports are counted once, not summed: collection is replicated on every worker. Missing or disagreeing reports, or worker crashes, produce status incomplete with reason code worker_reports_incomplete and null counts. This is a reporting uncertainty, not a claim that the original suite ran or a change to selection or xdist's handling of failures. Module mode's early empty exit is reported by the controller before workers start.

Reports are written at session finish, including failing test runs, or before module mode's early empty-selection exit (status 5). The destination's parent must already exist; existing files are overwritten. Write failures emit a PytestWarning rather than replacing pytest's outcome (normal pytest warning filters still apply). Without --skippy-cov, the report option has no effect: no file or summary is emitted. These additions do not change the standalone CLI or GitHub Action interfaces, selection scopes, or exit codes.

Coverage baseline

Run skippy-cov from any directory inside a Git checkout. Diff paths and source inspection use the actual Git checkout root (not pytest's configuration root). Without Git, the current directory is used.

User-supplied coverage-file paths (including the default .coverage), patch-file paths, --source-root, and explicit test scopes remain relative to the invocation directory. CLI output is also invocation-relative; --strip-prefix instead strips the requested --relative-to scope.

By default, relative coverage filenames and pytest context paths are interpreted relative to the checkout root, regardless of the invocation directory. For a relative_files = true baseline recorded from a checkout subdirectory, explicitly set --source-root packages/api (pytest: --skippy-cov-source-root packages/api) when invoking from the checkout root, or --source-root . when invoking from packages/api. This rebases both source records and test contexts into that checkout subdirectory. The recording setup must use the same base for relative source and context paths; differing bases are unsupported.

For absolute relocated artifacts, a source root outside the current checkout is the original checkout root, whose prefix is replaced by the current checkout root. Absolute records under an in-checkout source root retain that subdirectory prefix too. Unmappable absolute records are rejected.

Relative coverage contains no recording-directory metadata. Neither the current invocation directory nor basename matches establish its origin. A missing exact match with a relative suffix in the baseline raises an actionable recording-root error (the plugin falls back to the original selection), never a guessed match. This cannot detect every mismatch: colliding paths may be valid under either origin. You must provide the recording directory explicitly for subdirectory baselines, even if a same-named file exists at the checkout root.

Pytest stays in the invocation directory: configuration discovery, default collection, -k, -m, and exact node IDs retain their normal pytest semantics. Selection never expands the caller's scope. The pytest plugin warns and keeps the original test selection when baseline validation fails.

The baseline must be a readable coverage database with pytest test contexts, generated using --cov-context=test. Missing, empty, corrupt or context-free data is an error; the CLI exits with status 2 rather than printing an apparently valid empty selection.

Validation does not establish baseline completeness or revision provenance. The caller must use data from a successful complete test run, retain its commit SHA, and compare against that revision, including intervening changes to the base branch. A readable database alone does not guarantee a trustworthy selection.

Prefer portable coverage paths when collecting data in another checkout or Docker:

[tool.coverage.run]
relative_files = true
core = "ctrace"

If a baseline already contains absolute paths from another checkout, provide its original root explicitly:

skippy-cov --coverage-file baseline/.coverage --source-root /fierro
pytest --skippy-cov --skippy-cov-coverage-file baseline/.coverage --skippy-cov-source-root /fierro

Paths are matched relative to the checkout root, never guessed from basenames. Every recorded source path must map to that root. Mixed baselines from multiple absolute checkout roots are rejected rather than silently losing test consumers; combine them with coverage's path aliases or collect relative paths first. On Python 3.14+, coverage defaults to sysmon, which does not support dynamic test contexts; collect the baseline with core = "ctrace" as shown above.

Conservative selection and fallback

By default, selection runs whole test modules. Pytest discovers unittest classes, inherited methods, custom function names and parameterizations; skippy-cov does not rebuild their node IDs with a separate AST visitor.

Individual test selection (opt-in)

The default granularity is module. Individual selection is an opt-in heuristic: caches and state shared between tests can hide consumers from coverage. Keep module selection as the default and run the complete suite before merging. To select individual tests, use:

pytest --skippy-cov --skippy-cov-diff=BASELINE_SHA..HEAD --skippy-cov-granularity=test
skippy-cov --diff BASELINE_SHA..HEAD --granularity test --format json

The coverage database must have been collected at BASELINE_SHA, with --cov-context=test. Its revision cannot be inferred from a coverage database alone; providing coverage from a different commit can produce an incorrect selection. A single Git ref uses its merge base with HEAD, while an explicit BASELINE_SHA..HEAD range uses that exact baseline. The new revision must be HEAD. With a patch file, provide --base-ref BASELINE_SHA to the CLI or --skippy-cov-base-ref=BASELINE_SHA to pytest so the old source is available.

This mode parses function, async-function, method, nested-function, and class ranges from the old source without executing it. A function-body change selects the tests recorded anywhere in that function's baseline runtime body, including consumers in other test files. It uses old line numbers for coverage, so shifted definitions and insertions into previously uncovered branches remain eligible. Unrelated sibling functions and methods do not share a selection. Decorated functions and methods inside decorated scopes require the original selection: a decorator such as functools.lru_cache can return a value without executing the recorded function body for every consumer.

Every new test is included independently of coverage and source impact. Changed test modules run in full, including new functions, inherited methods, and new parameter cases. Removing lines without known runtime coverage, including parameter declarations, requires the original selection. The plugin keeps pytest's original collection and then deselects unaffected tests, respecting requested paths, exact node IDs, -k, and -m. For an affected parameterized test it runs all currently collected cases within that original scope. The CLI resolves current IDs through an isolated pytest collection; pytest must be installed for this mode. It keeps stdout suitable for the selected output format and protects the baseline coverage database during collection. Test paths/discovery configuration apply; the CLI's collection clears execution addopts and PYTEST_ADDOPTS. Apply execution filters to the eventual pytest invocation, or use the plugin to keep the full original pytest argument semantics.

For independent suites, repeat --collect-group to collect each group in a separate Python process. All groups are combined before impact selection. Pass collection options explicitly, including options normally set in addopts:

skippy-cov --diff BASELINE_SHA..HEAD --granularity test --format json \
  --collect-group tests/client tests/shared --collect-group tests/server \
  --pytest-arg=--import-mode=importlib

Each group accepts one or more paths; use the resulting node IDs with the original suite commands. --relative-to still filters the final output, so omit it when all groups must be represented. Pytest arguments apply to every group; configuration addopts and PYTEST_ADDOPTS remain cleared. Pass only arguments needed for collection, without execution or coverage options. A failed group or unsafe impact exits nonzero rather than returning a partial selection; an empty JSON object is a successful empty selection. Collection imports test modules, but does not run tests or fixtures, and uses a temporary coverage database.

New standalone functions, signature/decorator changes, class attributes/bases, module declarations, ambiguous scope matches, unknown/import-time coverage, and shared setup/teardown require the original selection. Recorded file consumers cannot establish every import-time dependency, such as a constant imported by another test module. The plugin preserves the original selection with a diagnostic; the CLI reports that it cannot select safely. New source files and non-Python changes retain that full-selection fallback. The test suite compares individual, module, and full runs with real coverage, including a shared-session-fixture regression where narrowing could hide a failing second consumer. Historical coverage remains an impact heuristic.

Module selection rules

Changes to existing source function bodies with recorded execution under a test's run phase select the consuming test modules. If a source diff also changes imports, declarations, or otherwise has unknown impact, skippy-cov preserves the original selection. Shared fixtures may execute only for the first test that requests them, so those contexts cannot identify every test that reuses their results. A file-level fallback involving setup/teardown therefore requires the original pytest selection, as does a file with no recorded consumers. This does not prevent selection for validated function-body changes covered only during run in the same source file.

Changes to test modules select the whole module when edits stay within test functions, including newly added tests and pytest.mark.parametrize cases. Removing lines with unknown or import-time coverage requires the original selection; comments and blank-looking text inside string literals are code. Imports, other decorators, fixtures, and recorded setup/teardown execution still require the original selection because they can affect shared test setup. Conftest files, non-Python inputs, new/deleted/renamed files, and unknown coverage also require the original selection. Recorded consumers of inherited tests are included alongside the edited module. Separate edit blocks are validated independently, even when Git groups them into one hunk. An empty diff or no affected tests inside the caller's scope exits with pytest status 5, not a successful test run.

The pytest plugin warns and keeps the original arguments when the baseline or impact is unknown. The CLI exits with status 2 for those cases: callers must run their full test command instead of treating an error as an empty selection. Explicit node IDs and absolute paths passed to pytest retain their scope. An explicit --rootdir does not change the base used for selected module paths. Unresolvable Git refs also warn and preserve the original pytest selection.

Historical coverage is a heuristic, not a proof of all dependencies. Reuse data only from a successful complete run, keep its commit SHA alongside the artifact, and compare against that commit, including intervening changes to the base branch. Dynamic imports, cross-test state and shared fixtures still warrant periodic full runs and a complete suite before merging.

GitHub Actions

The composite action at .github/actions/select finds the newest accessible coverage cache from a successful full workflow on the target branch whose commit is an ancestor of both the checkout and target ref. It verifies the cached SHA, then returns baseline, selected, targets (JSON) and reason. A missing, invalid or unsafe baseline returns selected: false; the caller chooses its full workflow. An empty JSON object is a successful empty selection.

After all full-suite jobs have succeeded, combine their coverage databases and save the baseline directory in the full workflow:

- run: |
    mkdir -p .skippy-baseline
    cp .coverage .skippy-baseline/.coverage
    printf '%s\n' "$GITHUB_SHA" > .skippy-baseline/sha.txt
- uses: actions/cache/save@v4
  with:
    path: .skippy-baseline
    key: skippy-baseline-v1-${{ github.run_id }}-${{ github.sha }}

Save only on pushes to the target branch; the action also verifies that the whole workflow ultimately succeeded. Configure the PR job with actions: read and contents: read, a checkout with an origin remote, and the project environment with pytest and a version of skippy-cov containing this action's collection options. For example:

- uses: actions/checkout@v4
  with:
    fetch-depth: 1
- id: skippy
  uses: mgaitan/skippy-cov/.github/actions/select@<release-tag>
  with:
    base-ref: ${{ github.event.pull_request.base.sha }}
    base-branch: ${{ github.event.pull_request.base.ref }}
    workflow: .github/workflows/ci-full.yml
    cache-prefix: skippy-baseline-v1
    history-depth: 100
    collection-groups: '[["tests/client"], ["tests/server"]]'
    pytest-args: '["--import-mode=importlib"]'
    # source-root: /original/container/checkout
- if: steps.skippy.outputs.selected != 'true'
  run: echo "Full suite required: $REASON"
  env:
    REASON: ${{ steps.skippy.outputs.reason }}

For shallow checkouts, the action performs one git fetch --no-tags --depth=N from origin for the checked-out HEAD and target commit, where N is history-depth (default 100, a positive integer). Prefer the PR's exact base SHA as above; a target ref must otherwise be available locally or fetchable from origin. The fetched target is stored in refs/skippy-cov/base, without changing the checkout or local branches. Complete checkouts with an available target do not fetch. This bounds history depth per tip, not total transfer bytes (merge history and repository contents can still be large).

The action never automatically deepens again or downloads the full history. It selects a cache only when Git proves its revision is an ancestor of both HEAD and the target in the available history. An older baseline beyond the shallow boundary, unrelated history, or a failed fetch yields selected: false and the full-suite fallback, never an assumed ancestry. Increase history-depth if baselines are commonly older; fetch-depth: 0 remains an explicit option when full history is preferred.

Use the exact same baseline-path when saving and restoring: GitHub cache versions include the path. The cache prefix must identify a compatible test environment and complete suite. Capture per-test coverage with --cov-context=test for individual selection. Collection runs in the action's job environment, so install the project's test dependencies there or run the job in the appropriate container. Test execution, runner setup, service setup, and decisions about empty selections remain with the caller.

Contributing

See CONTRIBUTING.md for information on how to contribute to the project.

License

See LICENSE for licensing information.

Metadata

Release files for skippy-cov 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for skippy-cov 0.5.0
File Size Uploaded Provenance Actions
skippy_cov-0.5.0.tar.gz 69.9 kB With provenance Details for skippy_cov-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for skippy-cov 0.5.0
File Interpreter ABI Platform Actions
skippy_cov-0.5.0-py3-none-any.whl Python 3 none any Details for skippy_cov-0.5.0-py3-none-any.whl

Total release size: 95.7 kB

Release files / skippy_cov-0.5.0.tar.gz

Download URL skippy_cov-0.5.0.tar.gz
Size 69.9 kB
Tags Source
SHA-256 checksum
How to use checksums
5900df5a1b3b4e5a4df0d1204f70b09190bb06a9640a8883c1ca57b57c3d53b3
BLAKE2b-256 checksum
How to use checksums
0b822eae5d5d72fdf4910da33159a2096058555457975db9f442fe98b3f5b87d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Release files / skippy_cov-0.5.0-py3-none-any.whl

Download URL skippy_cov-0.5.0-py3-none-any.whl
Size 25.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0945ea7a21d9eb264999fef4eb624c8fa2f23f116857ce0f0defbbf927d9dfd8
BLAKE2b-256 checksum
How to use checksums
586fd45a30fec511d9d2cc9d93e6c3aed1b93fb9a1230887195bd54ab5385da2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Security for skippy-cov 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for skippy_cov-0.5.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page