mutmut-ratchet

Shared CI tooling that makes a slow mutmut run usable as a blocking pull-request gate: scope the run to what a PR touched, fan it across a time-balanced matrix, and fail only on a real per-file mutation-score regression.

It is the de-duplicated form of the scripts/mutation_*.py helpers that were copy-pasted between rtl-433-hass/rtl_433 and rtl-433-hass/pyrtl_433. Only the per-repo constants differed; those are now configuration.

Why

A full-package mutmut run is slow (tens of minutes), and its score is not perfectly reproducible. Naively gating on it either blocks every PR for an hour or fails spuriously. The five subcommands fix that:

Subcommand	What it does
targets	Turns git diff --name-only into the modules worth mutating, or escalates to a full run
shards	Deterministic LPT partition of the package into N time-balanced shards
stats	Reduces mutmut's on-disk mutants/*.meta into a comparable JSON payload
ratchet	Compares that payload to a committed baseline with a mutant-denominated tolerance band
timings	Records the measured per-file mutmut runtime that shards balances on

Two design choices carry most of the value:

• Shards balance by measured time, not mutant count. Per-mutant test time varies several-fold across modules, so a count-balanced split still leaves a slow pole that the gate waits on. Modules are sorted heaviest first (ties by path) and dropped into the lightest bin (ties by lowest index) — plain LPT, within 4/3 of optimal makespan and fully reproducible, so every matrix job computes the identical assignment with no coordination.
• The ratchet's tolerance band is denominated in mutants. A flat percentage is wrong in both directions: 2% is ~13 mutants on a 630-mutant module but zero mutants on a 29-mutant one. The band is max(tolerance_fraction × total, tolerance_mutants) mutants, converted back to score space. That absorbs both run-to-run noise and the scoped-vs-full gap (a scoped run is a slight lower bound: some mutants are only killed by tests in other files) while still failing on a genuine regression, which typically costs far more mutants than the band. The baseline only ever ratchets upward.

Install

uv add --dev mutmut-ratchet

mutmut itself is a dependency (the tooling reads mutmut's module walk and its per-file meta through mutmut.__main__), so installing this pulls it in. Pin the exact mutmut version you test against in your own dev group if you care which one CI uses.

Configure

All settings live in your pyproject.toml under [tool.mutmut_ratchet]. Only package_path is required; unknown keys and wrong types are hard errors, so a typo can never silently weaken the gate.

[tool.mutmut_ratchet]
package_path = "my_package"
escalate_paths = ["pyproject.toml", "uv.lock", "tests/conftest.py"]

[tool.mutmut_ratchet.explicit_test_sources]
"tests/test_urls.py" = ["_urls.py"]
"tests/test_mut_client_floor.py" = ["client.py"]

Key	Type	Default	Meaning
package_path	string	(required)	Repo-relative path of the package under mutation, e.g. pyrtl_433 or custom_components/rtl_433.
package_dotted	string	package_path with / → .	Dotted name mutmut uses in mutant filters. Set it only when the two differ (e.g. a src/ layout).
baseline	string	"scripts/mutation_baseline.json"	Committed per-file score baseline. Relative paths anchor to the pyproject.toml's directory.
timings	string	"scripts/mutation_timings.json"	Committed per-file mutmut runtime profile, used as the shard bin-pack weight.
escalate_paths	list of strings	["pyproject.toml", "tests/conftest.py"]	Changed paths that force a full run because they can move results package-wide. Exact repo-relative matches.
explicit_test_sources	table of string → list of strings	{}	Test file → the source modules (relative to package_path) it exercises, for tests the test[_mut]_<name>.py → <name>.py convention cannot resolve. Genuinely broad tests are deliberately left out so they still escalate.
tolerance_fraction	float	0.02	Fractional half of the band; scales the cushion on large files.
tolerance_mutants	int	3	Absolute half of the band, in mutants; protects small files where a percentage rounds to nothing.
precision	int	6	Decimals scores are rounded to before comparison.
floor	float	0.70	Advisory floor recorded into a freshly written baseline payload.
fallback_seconds_per_mutant	float	1.0	Weight used when neither a timing nor any timing profile exists at all (a fresh checkout).

A committed baseline carries the tolerance_fraction / tolerance_mutants it was written with, and those win over the config defaults — so retuning the band is a baseline edit, not a workflow edit. An explicit --tolerance-* flag beats both.

Precedence throughout is CLI flag → pyproject.toml → built-in default.

Commands

Every subcommand also accepts --config PYPROJECT (default: the nearest pyproject.toml at or above the cwd), --package-path, and --package-dotted.

mutmut-ratchet targets [--base REF] [PATH ...]

Maps changed paths to targets. With no arguments, reads whitespace-separated paths from stdin. Prints four lines: all or scoped; the mutmut filter patterns; the source paths; and the functions the scope was narrowed to (empty when nothing was). Exit 0.

--base REF narrows the scope to the changed functions rather than whole modules, by diffing against REF and mapping each hunk to the function that contains it. A one-line edit to a 750-line module then mutates one function instead of all of it. Three rules keep that sound:

• A changed line belonging to no mutable function — module-level code, an import, a class body, a nested class's method — escalates that file to its whole module, because such a change can affect anything in the file.
• A source pulled in because one of its tests changed is never narrowed: a weakened test can free a mutant anywhere in the module it exercises.
• No line information at all (no --base, or git cannot answer) falls back to whole modules, never to an empty filter.

Line 4 feeds shards --restrict-functions. A module that stays whole contributes nothing to it — the shard falls back to that module's own pattern when no name mentions it — so this never has to enumerate a file's functions.

The diff is only taken once the scope is known to be narrowable, and only over the sources in it, so an escalated or docs-only PR never pays for it.

Function-scoped runs need --mode functions to gate; see below.

Patterns are derived the same way shards derives them, so a package __init__.py in scope emits the x_* / xǁ* trampoline patterns its mutants are actually named after — mutmut drops the __init__ segment from mutant names, so a <pkg>.sub.__init__.* filter would match none of them and silently run zero mutants.

mutmut-ratchet shards --shard N --of M [--restrict PATH ...] [--restrict-functions NAME ...] [--baseline P] [--timings P]

Prints two lines: this shard's filter patterns and its source paths (both blank for an empty shard). --restrict intersects the shard with a scoped set without changing the global assignment, so a scoped PR fans across the same shards. --restrict-functions (line 4 of targets --base) narrows the patterns further, to those functions; a module in the shard that none of them name keeps its whole-module pattern. Exit 0, or 2 for out-of-range --shard/--of.

mutmut-ratchet stats [--paths PATH ...]

Prints the stats JSON on stdout, with a files block and a functions block ({path: {function: tally}}). --paths is required after a filtered mutmut run: mutants outside the filter stay "not checked", which would otherwise read as 0%. Exit 0.

Every tally also reports not_checked — the subset of survived that mutmut recorded but never executed. That is what a function-scoped run needs: no list of what the run covered has to be threaded in from outside, because a function whose mutants are entirely not-checked was plainly not measured, and --mode functions skips it. It is also how a partial file tally is kept out of the baseline.

mutmut-ratchet ratchet --mode floor|strict|functions --stats FILE [--baseline P] [--update] [--tolerance-fraction F] [--tolerance-mutants N] [--tolerance-survivors N]

floor is the CI gate (improvements never fail); strict also fails on upward drift, as a local check that the committed baseline is still representative. --update writes current scores back, keeping the higher of baseline/current per file, and refuses while regressions stand. Exit 0 pass, 1 regression/drift, 2 missing stats or baseline.

functions is the gate for a function-scoped run. A per-file score is a partial measurement when only some of a file's functions were mutated — the rest come back "not checked", counted as survived — so it cannot be compared against a whole-file baseline. Each mutated function's score can. Two rules:

• A function already in the baseline fails only when it gains survivors beyond --tolerance-survivors (default 0). Counting survivors rather than comparing scores is what makes this no new survivors: a function that grows by ten well-tested lines keeps its count, and touching a function that already had survivors does not block the PR.
• A function not in the baseline must reach floor, since there is nothing to compare against and new untested code is the case the gate exists for.

The default tolerance is 0 — no new survivors means none. The per-file band exists because a file's score is compared across runs and across the scoped/full divide, where a mutant or two genuinely moves; this gate counts survivors in the handful of functions a PR changed, where that does not apply.

The one mechanism that looks like noise is worth being precise about: mutmut counts a timeout as a kill, but its budget is (estimated_test_time + timeout_constant) × timeout_multiplier — 15× plus a second, by default. A mutant must run an order of magnitude longer than the tests it runs under to trip that, which is an unbounded loop rather than a busy runner, and an unbounded loop trips it every time. What could still flip is a genuinely nondeterministic test, and a band would only hide that in the functions a PR is actively changing. Raise it per repository if a suite really does flap, and treat needing to as the finding it is.

Rolling this onto an existing baseline is a bootstrap: with --update, a baseline that has no functions block records one without gating, and every run after that gates against it.

mutmut-ratchet timings [--out PATH]

Writes the runtime profile after a full mutmut run. Exit 0, or 2 when no timing data exists yet.

Worked GitHub Actions example

Scoped on PRs, full on push/schedule, sharded six ways either way, with one aggregated status check.

name: Mutation

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 3 * * *"

permissions: {}

jobs:
  mutation:
    name: "Mutation floor — shard ${{ matrix.shard }}"
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v5
        with:
          # Need history back to the base branch to diff a PR's changed files.
          fetch-depth: 0
      - uses: astral-sh/setup-uv@v6
        with:
          enable-cache: true
      - run: uv sync --locked --dev

      - name: Determine scope for this trigger
        id: scope
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            git fetch --no-tags origin "${{ github.base_ref }}"
            changed=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")
            uv run mutmut-ratchet targets $changed > targets.txt
            cat targets.txt
            {
              echo "mode=$(sed -n '1p' targets.txt)"
              echo "paths=$(sed -n '3p' targets.txt)"
            } >> "$GITHUB_OUTPUT"
          else
            echo "mode=all" >> "$GITHUB_OUTPUT"
            echo "paths=" >> "$GITHUB_OUTPUT"
          fi

      - name: Determine this shard's modules
        id: shard
        run: |
          if [ "${{ steps.scope.outputs.mode }}" = "all" ]; then
            uv run mutmut-ratchet shards --shard ${{ matrix.shard }} --of 6 > shard.txt
          elif [ -n "${{ steps.scope.outputs.paths }}" ]; then
            uv run mutmut-ratchet shards --shard ${{ matrix.shard }} --of 6 --restrict ${{ steps.scope.outputs.paths }} > shard.txt
          else
            # Scoped run with nothing in scope (e.g. a docs-only PR): no work.
            printf '\n\n' > shard.txt
          fi
          {
            echo "patterns=$(sed -n '1p' shard.txt)"
            echo "paths=$(sed -n '2p' shard.txt)"
          } >> "$GITHUB_OUTPUT"
          cat shard.txt

      - name: Run mutation testing (this shard's modules)
        if: steps.shard.outputs.patterns != ''
        run: |
          uv run mutmut run ${{ steps.shard.outputs.patterns }}
          uv run mutmut-ratchet stats --paths ${{ steps.shard.outputs.paths }} > mutation-stats.json

      - name: Enforce per-file floor vs baseline
        if: steps.shard.outputs.patterns != ''
        run: uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json

      - name: No modules in this shard
        if: steps.shard.outputs.patterns == ''
        run: echo "No modules in scope for this shard; nothing to check."

  mutation-gate:
    name: "Mutation floor"
    needs: [mutation]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Aggregate shard results
        run: |
          if [ "${{ needs.mutation.result }}" != "success" ]; then
            echo "One or more mutation shards failed."; exit 1
          fi
          echo "All mutation shards passed."

A simpler single-job form, for a package small enough not to need a matrix:

      - name: Mutation test (per-module floor ratchet)
        run: |
          uv run mutmut run --max-children 4
          uv run mutmut-ratchet stats > mutation-stats.json
          uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json

Never cache the mutants/ working tree: a stale cache can report a higher score than reality and let a regression through. Caching uv's download cache is fine.

Bootstrapping a baseline

uv run mutmut run                                   # a FULL run
uv run mutmut-ratchet stats > mutation-stats.json
uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json --update
uv run mutmut-ratchet timings                       # writes the shard weights

Commit both JSON files. Refresh the timings profile periodically; a stale one degrades to a slightly suboptimal — never incorrect — split.

Licence

Apache-2.0. See LICENSE and NOTICE.