covopt 0.1.0a0

Budgeted local test selection

covopt

covopt is a small command-line tool for budgeted local test selection.

It reads a diff from stdin, loads a precomputed model describing which tests are informative for which parts of the repository, and emits a subset of tests that maximizes expected signal under a runtime budget.

The intended workflow is:

git diff | covopt select -t 10 | xargs pytest

This is not a full test runner and does not integrate with pytest directly. Its only job is to:

1. parse changed code from a diff,
2. map those changes into an internal repository-space representation,
3. score candidate tests by expected value,
4. penalize redundant tests,
5. choose a subset that fits a wall-clock budget,
6. print selected test IDs to stdout, one per line.

Motivation

Large test suites often contain many tests that are highly correlated. For local iteration, running all tests is too expensive, while running only “affected” tests can still waste time on near-duplicates.

covopt treats test selection as a budgeted optimization problem:

• Return: expected defect-detection signal in changed areas
• Cost: estimated runtime of a test
• Risk: residual untested change exposure
• Correlation: redundancy between tests

The goal is to select a small, diverse, high-value subset of tests for fast feedback, while leaving full validation to CI.

Core idea

The repository is modeled as a feature space over code regions such as files, modules, classes, or functions.

Each test has:

• a sparse signal vector over that space,
• an estimated runtime cost,
• optional metadata such as historical failures or flakiness.

A diff is converted into a weighted change vector over the same space.

Selection then solves:

• maximize coverage of changed regions,
• prefer tests with high marginal gain per second,
• avoid picking multiple tests with nearly identical signal,
• stop when the budget is exhausted.

In practice this is implemented as a budgeted greedy optimizer over a diminishing-returns objective.

Non-goals

For the first version, this project does not aim to provide:

• direct pytest plugin support,
• dynamic collection of coverage data,
• mutation testing,
• distributed execution,
• CI orchestration,
• perfect safety guarantees.

This is a local feedback accelerator, not a replacement for the full test suite.

CLI

Basic usage

git diff | covopt select -t 10

Prints selected test node IDs to stdout.

Example

git diff HEAD~1 | covopt select -t 10 | xargs pytest

Arguments

usage: covopt select [-h] -t SECONDS [-n N] [--verbose]

Optional

• -n N Restrict optimization to the top N candidate tests after initial scoring.

• --verbose Emit diagnostics to stderr.

Input / output contract

Input

stdin must contain a unified diff, for example from:

git diff
git diff HEAD~1
git show <commit>

Output

By default, the tool prints one test per line:

tests/unit/foo/test_parser.py::test_basic_parse
tests/unit/bar/test_config.py::test_defaults
tests/integration/api/test_health.py::test_healthcheck

This makes it easy to pipe into xargs pytest.

With --verbose, the tool emits structured output including estimated total runtime and selection scores to stderr. Errors are always written to stderr.

Model format

The selector consumes a model containing:

• repository feature definitions,
• per-test sparse signal vectors,
• pairwise or cluster-level redundancy information,
• test runtime estimates,
• optional weights and calibration parameters.

A minimal conceptual schema looks like this:

{
  "features": [
    "pkg.module_a",
    "pkg.module_a.fn_x",
    "pkg.module_b",
    "pkg.module_b.ClassY.method_z"
  ],
  "tests": {
    "tests/unit/test_a.py::test_one": {
      "cost": 0.8,
      "signal": {
        "pkg.module_a": 0.9,
        "pkg.module_a.fn_x": 1.0
      }
    }
  }
}

The model is stored in a .covopt file that is generated when invoking the tool.

The exact on-disk format is an implementation detail and may evolve.

Selection algorithm

At a high level:

1. Parse the diff from stdin
2. Map changed files / hunks / symbols to repository features
3. Build a weighted change vector
4. Score candidate tests by overlap with the change vector
5. Iteratively pick the test with the best marginal gain / cost
6. Apply diminishing returns to already-covered regions
7. Penalize redundant tests using similarity
8. Stop when the next test would exceed the budget

A typical utility function is:

utility(S) = sum over features j of w_j * f(sum over tests i in S of A_ij)

Where:

• w_j is the change weight for feature j
• A_ij is the signal of test i on feature j
• f(...) is a saturating function so duplicate tests add less value

This makes the selector naturally prefer diverse tests over repeated variants of the same test shape.

Project structure

covopt/
  __main__.py         # CLI entrypoint
  cli.py              # argv parsing and I/O orchestration
  diff_parser.py      # unified diff parsing
  feature_space.py    # repo feature mapping
  scoring.py          # initial candidate scoring
  optimize.py         # budgeted greedy selection
  output.py           # stdout / json formatting

Exit behavior

• 0: successful selection
• non-zero: invalid arguments, malformed diff, or internal failure

Diagnostics should go to stderr. Selected tests should go to stdout only.

Design principles

• Fast startup: suitable for local shell pipelines
• Deterministic: same diff => same selection
• Composable: works well with Unix pipes
• Model-driven: selection logic is decoupled from model construction
• Conservative stdout: only emit test IDs unless JSON mode is requested

Example workflow

git diff | covopt select -t 10 | xargs pytest

Full CI still runs the complete suite:

pytest

Future work

• richer repository feature extraction
• learned test-value calibration from historical failures
• improved redundancy modeling
• coverage/model builders
• optional pytest integration
• support for multiple optimization strategies

Status

Early-stage experimental project focused on the core algorithm and CLI. The first milestone is a reliable selector that can consume a diff, apply a time budget, and emit a useful non-redundant subset of tests for local development.