crucible-harden 0.1.0

Adversarial test-hardening loop for AI-built code: Tester vs Critic closing mutation feedback, mechanical verdicts, metered spend, receipts.

crucible

Your AI wrote the tests. Who tested the tests?

_{Recorded live — every command really ran, and every number in it is read back from that run's own receipt. Sped up for viewing; replay the raw cast at real speed. Full landing page: jott2121.github.io/crucible}

That module has 7 passing tests and 97% line coverage. Mutation testing injects 71 real defects. The suite kills 46. 25 survive — twenty-five real bugs walking straight through a green build.

crucible kills 24 of the 25, and then does the more important thing: in two rounds the Critic wrote tests that failed on pristine code, and crucible threw them out rather than bank a kill it hadn't earned. The loop ends dry with one mutant still standing, and says so. Mutation score 65% → 99%; line coverage never moves off 97% the entire time. Coverage was never the thing measuring your safety.

An earlier run on the same module killed all 25 (clean). Same tool, same module, different day — model nondeterminism is real, and the receipts record both rather than only the flattering one.

Put the number in your own PRs

The diagnose costs nothing and calls no model. Point the Action at a module and every pull request tells you how many real bugs your suite would actually have caught:

- uses: Jott2121/crucible@v1
  with:
    module: yourpkg/yourmodule.py    # omit if your repo already configures [tool.mutmut]
    fail-under: "70"                 # optional: red the build below this

It comments the score and names the survivors. No model, no API key, $0 — the number that embarrasses a coverage badge is free to compute, and you should not need a subscription to be told the truth about your own tests.

For the badge, set badge-file: mutation.json, publish it, and point shields.io at it. The payload lives in your repo — there is no badge service of mine sitting in the middle of your CI, and no uptime I owe you. The badge at the top of this page is generated exactly that way, by this Action, on this repo:

[![mutation](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/YOU/REPO/badges/mutation.json)](...)

Or just run it locally, on any repo, right now:

crucible score . --module yourpkg/yourmodule.py --coverage 97
97% line coverage, but 25 of 71 injected defects SURVIVED this suite (46 killed, mutation score 65%).

Coverage measures what ran, not what would be caught. Mutation testing injects real defects and counts how many your suite kills — and AI-written suites routinely leave survivors. crucible closes the loop: a Tester agent writes tests, mutmut finds the survivors — injected defects no test caught — and a Critic agent is handed the named survivors and writes tests to kill exactly those. Every verdict is mechanical: pytest kills the mutant or it survives. No model ever grades model output.

Quickstart — the free first win (no model, no keys, one command)

Find out what your existing tests actually miss, on your own repo:

pip install crucible-harden       # the command it installs is `crucible`

cd /path/to/your-repo-clone       # work in a clone: crucible writes scope config
crucible score . --module yourpkg/yourmodule.py --coverage 97

97% line coverage, but 25 of 71 injected defects SURVIVED this suite (46 killed, mutation score 65%).

No model is called and no API key is needed. crucible score detects your layout, writes the mutation scope, and runs mutmut on it. For the airtight pre-flight check — a $0 canary that refuses (exit 4) unless a freshly written test actually collects and registers a kill under your scope — run crucible scope first (the bundled harden ritual does). If your repo already configures [tool.mutmut], drop --module and crucible grades the scope you chose instead of overwriting it.

The survivor count is plain mutation testing on your own suite. No AI is involved yet — the number that embarrasses a coverage badge is free to compute.

Then harden — the adversarial loop

crucible harden . --module yourpkg/yourmodule.py \
    --tester claude-cli --critic claude-cli --runs-dir ~/.crucible-runs/yourrepo

With claude-cli, model calls run through Claude Code headless on your Claude subscription — $0 metered spend, and every run's meta.json records billing: max-plan so plan-covered shadow dollars are never mistaken for an invoice. No subscription? --tester anthropic uses the metered API via ANTHROPIC_API_KEY.

Lean invocation is the default: the subprocess runs with --tools "", collapsing Claude Code's agent loop to a single completion. On the reference run (rag_guard/guard.py), that took the harden from 439,230 to 3,641 input tokens (120.6×), with identical results — the same 25/25 surviving mutants killed (receipts 20260712T050833Z vs 20260712T171312Z). Measured on that run's receipts, not a universal constant. CRUCIBLE_LEAN=0 restores the ambient invocation.

Receipts are the product

Every run writes a receipt directory:

meta.json         # models, billing (api vs max-plan), lean_isolation rung, scope commit
receipt.jsonl     # one line per round: tokens in/out, cost, kills, survivors, prompt hash
result.json       # verdict + totals

Generated tests are written into the working tree of wherever you run it — so run it on a throwaway branch; the bundled harden-tests skill (.claude/skills/harden-tests/) enforces the full ritual: local branch only, never main, PR strictly opt-in. If the canary can't prove your scope, crucible refuses instead of spending tokens.

Results

Two pre-registered experiments (experiments/PROTOCOL.md, experiments/PROTOCOL-B.md). Experiment 1 ran five subjects across three arms (one-shot, same-lineage adversarial loop, cross-lineage adversarial loop). The loop-arm pipeline outscored one-shot 105 discordant kills to 0 (pooled exact McNemar p = 4.9×10⁻³²) — a pipeline-level replication of the direction established by MuTAP, AdverTest, and Meta's ACH (docs/RELATED-WORK.md); because each arm drew its own round-0 suite, it is not a causal estimate of the loop, a confound an external cross-model review of the draft paper caught. Experiment 2 removed it: one frozen round-0 suite per replicate (5 per subject × 4 subjects), committed before its continuations, all three arms run from that identical state. Within that design the Critic loop's incremental effect is causal and large: a mean 78% of the survivors the frozen round-0 left standing were killed by the critic rounds (rate 0.783, 95% bootstrap interval [0.592, 0.935]). The cross-provider critic configuration — pre-registered as a no-verdict pilot — showed a positive within-replicate difference (rate gap 0.178 [0.039, 0.347]; direction stable under every leave-one-out sensitivity, magnitude 59% driven by one receipted truncation failure) at 5.5× lower arm cost. An earlier analysis had shown an apparently overwhelming cross-lineage effect; the autopsy traced it to silent output truncation deleting one arm's rounds — an instrument artifact, not a model difference — and the same mechanism recurred three times in Experiment 2, this time mechanically detected and honestly scored. That autopsy and the fail-closed instrumentation built from it are the finding. Full tables: experiments/RESULTS.md (Experiment 1), experiments/RESULTS-B.md (Experiment 2), and the paper draft with its complete cross-model review trail in paper/.

Why trust this

The claims above are checkable: both experiments were pre-registered before results existed (Experiment 2's protocol was pushed publicly before its runner was even built), and inconclusive or corrected results are published at the same prominence as positive ones — including a manufactured effect the instrument itself produced, autopsied with receipts. Additionally, the prior art is cited rather than rediscovered (docs/RELATED-WORK.md), and the tool is dogfooded — crucible's own modules run under the same mutation gate, current score and survivor dispositions in docs/MUTATION.md.

How is this different from asking Claude to improve tests with mutmut?

Honest answer first: the core loop is the same, and you can hand-roll it. A model writing tests against mutmut's survivor list is exactly what a good one-file workflow does, and for strengthening a single suite on your own machine that workflow is genuinely fine. crucible is not a different algorithm. What it adds is the discipline that makes the result trustworthy to someone who was not watching it run:

• It refuses to spend a token on a scope it has not proven is real. crucible scope runs a $0 canary probe before any model is called: it proves the scope actually generates mutants and that a freshly written test really gets collected and registers a kill — or that your existing suite already kills under it. A scope it cannot prove is a refusal (exit 4), not a run, and the bundled harden ritual runs this gate first — so the failure where a model happily writes tests nothing ever executes burns zero tokens instead of a budget.
• The loop is iterated, not a single pass. A separate Critic re-attacks whatever is still surviving, round after round, until dry or a cap. On the frozen round-0 experiment (same-lineage arm, 4 subjects × 5 replicates) the critic rounds killed a mean 78% of the survivors a single generation left standing (0.783, 95% bootstrap interval [0.592, 0.935]) — measured, not asserted (full tables).
• It is strictly add-only. Generated tests are only ever added; an assert_add_only guard trips if a run would edit or delete anything already in your suite. (crucible does write a [tool.mutmut] scope into the clone's pyproject.toml and commit it — that commit is exactly what the receipt binds to.) With the local-branch-only ritual, the blast radius on your existing tests is zero.
• Kills you did not earn do not count. A generated test that fails on unmutated code is dropped test-by-test — the rest of the file is salvaged — and every kept file has to pass on pristine code twice, so a flaky kill is rejected rather than banked. A suite that is red on pristine code stops the run before a model is ever called (mutmut refuses a red baseline too — crucible just refuses before you have paid for the generation).
• Every run is a receipt bound to a commit. Models, tokens, cost, verdict, and prompt hashes are written to disk and tied to the scope commit, so a result is auditable by someone who was not in the room — which commit, which models, which prompts, what it cost — rather than a summary you take on faith.

If none of that matters for your use — one suite, one machine, one pass — a one-file workflow will serve you well. crucible earns its extra surface exactly when the number has to survive being handed to a teammate, a CI gate, or a reviewer.

Honest limitations

• Python + pytest repos only; layout heuristics target well-formed projects — a repo the canary can't validate is a refusal, not a guess.
• mutmut is pinned exactly (3.6.0): the src-layout shim relies on a mutmut-internal contract.
• The claude-cli provider has no mechanical truncation check (the CLI exposes no output cap); disclosed in the provider docstring.
• The 120.6× lean result is one module, one apples-to-apples pair of runs. Your ratio will differ; your receipts will tell you.

How it works

Tester (writes tests) ──> mutmut (injects defects, counts kills)
      ^                        │ named survivors
      └──── Critic (kills exactly those) <──┘   ... until dry or round cap

Built on oracle-gate (survivor triage, provenance, providers). MIT license.

Companion instrument: quorum-cal — measures how many independent votes an LLM judge panel is actually worth. Its truth-by-construction gold sets are built from exactly the killed mutants crucible produces (measured: three identical judges are ~1.3 effective votes; diverse panels shared zero false endorsements of real bugs).

Companion instrument: sabot — the same question one level up. crucible asks whether your tests would notice a planted defect; sabot plants a fault inside a running multi-agent pipeline and asks whether the pipeline's own reviewer and guardrail stages notice. Measured across LangGraph, CrewAI and AutoGen/Magentic-One, the median own-check detection rate was 16.7%, and the most common outcome was a correct answer with nothing flagging the fault.