TestGuard

Proves that a test suite actually defends the claims a project makes — by injecting the faults those claims say cannot happen, and reporting every fault the tests fail to detect.

Not a test generator. A claim verifier. Test generation is what happens after a claim turns out to be unfalsifiable.

Third tool following the Guard pattern, alongside docguard-cli (docs ↔ code) and websec-validator (attack surface ↔ code). All three run one loop:

declare what must be true → try mechanically to falsify it → freeze a baseline → gate only the delta → brief the agent before it writes code.

Why

Coverage cannot tell a test that pins correct behaviour from one that pins a defect. An agent that writes both the code and its tests encodes whatever it believed — including its bugs — and the suite goes green.

Measured on a real, entirely AI-authored production codebase with ~4,900 disciplined tests (no snapshots, 0.4% zero-assertion): 8 of 9 real historical bugs were invisible to the suite, worst case 2,451 tests green on known-broken code. The largest gap was a compliance-critical path with 100% coverage, where the one assertion that mattered used expect.objectContaining({...}) and omitted the field carrying the data.

A second, independent run on a different AI-authored codebase (63 test files, 458 tests, 24 hand-written security claims, 39 faults): 21 of 39 faults survived a fully green suite — 9 of them critical. Super-admin gating, membership checks, cookie flags and the whole authorization callback could be disabled without a single test noticing. One test file had re-implemented the authorization logic inside the test and asserted against the copy: fifteen green tests, zero detection. After wrapper-level tests were written against the survivors, 39/39 were killed.

The peer-reviewed picture in 2026 says the same thing from the other side. Coverage and mutation score of LLM-generated suites track effectiveness only when the code under test is assumed correct; once it may be buggy they "no longer serve as reliable indicators" (Zhao, Zhou and Cohen, ISSTA 2026). Buggy code steers a model toward tests that assert the bug, and prompting with the specification is the mitigation that works (arXiv 2607.22883) — which is why a claim here comes from intent, a fault is bound to the claim, and the brief hands the agent the claim before it writes. And agents saturate whatever tests they can see, with the gap to held-out tests growing about 28 points per tenfold increase in code size (SpecBench). Every generator on the market admits a test because it compiles, passes and raises coverage. TestGuard admits it because it fails when the claim is false.

Install

Projects that set min-release-age in .npmrc cannot see a version published less than that many days ago (ENOVERSIONS); install that one with npm i -D testguard-cli --min-release-age=0.

How it works

npx testguard-cli init        # install the agent layer: skill, session-start hook, AGENTS.md section
npx testguard-cli status --json   # where the project is and the ONE next action — the machine entry point
npx testguard-cli claims      # what does this project claim, and is every claim probeable?
npx testguard-cli probe       # try to falsify each claim; report what the tests missed
npx testguard-cli baseline    # freeze today's unproven findings; from now on only new ones gate
npx testguard-cli brief       # tell the agent where the suite is blind, before it writes
npx testguard-cli gate --changed origin/main   # fail when a changed source file carries no claim at all
npx testguard-cli scaffold src/x.ts   # propose faults for a file, as a draft to keep or drop
npx testguard-cli admit test/x.test.ts --claim X   # is this test green on HEAD and does it fail on every fault of X? ADMITTED or NOT ADMITTED

1. Claims live in testguard.claims.json (editors validate it against "$schema": "./node_modules/testguard-cli/spec/schemas/claims.schema.json"): a statement, where it comes from, which tests supposedly defend it, and one or more faults — each a deterministic source change that would make the statement false. Every claim and every fault records who produced it. testguard claims validates the file and reports drift against @claim <ID> annotations in source. Test files are deliberately not scanned — a claim asserted by a test is the authorship trap the tool exists for — and annotation ids must contain a hyphen so prose is never mistaken for one.

2. Probe confirms the defenders are green N times unmodified, applies each fault in a scratch git worktree (your tree is never touched), runs the defenders N times, re-runs survivors against the whole suite with N-run attribution (up to N: it stops as soon as no test has failed in every run, because from there none can be named a killer), restores, and classifies. It warns when another test runner is already running — a contended machine turns a slow suite into a TIMEOUT verdict about the load, not the claim — records that on the evidence, and offers --serial to run one test file at a time. Verdicts are a closed set:

   Verdict	Meaning
   killed	a test body rejected the behaviour, N/N — the only pass
   SURVIVED	the defenders stayed green while the claim was false
   NOCOVER	no test file defends the claim at all
   UNVERIFIABLE	the fault's anchor is missing or ambiguous — loud, never a skip
   TIMEOUT	the defenders hung; a hang is not a detection
   FAULT-INVALID	the replacement does not load — a bad fault, not a finding
   FLAKY-DEFENDER	the defenders are not reliably green, or disagreed across runs — detail.flakeRate says how often (failures of runs on unmodified source)

   Never a single score. Findings are ranked by severity, claim provenance and blast radius (relative imports, tsconfig path aliases and package.json#imports are resolved; bare package names are not), and written to .testguard/evidence.json — validated against the spec before it is written.

   Worktree mode probes a commit. If a defender or target file has uncommitted changes and you asked for the implicit HEAD, probe refuses and says so — otherwise your new tests would be silently absent and the same survivors would come back with no hint why. --include-dirty snapshots the working tree (tracked edits and new files) into a throwaway commit and probes that; your tree, HEAD and index are never touched. An explicit --ref (a pre-fix commit in a post-mortem, say) is honoured over a dirty tree, as is --ignore-dirty: a warning names the files and the evidence records them as repo.ignoredDirty, so the run says what it did not look at. Every summary names the commit probed.

   A claim with no defendedBy has its defenders discovered: the test files that import the fault's target, by relative path or resolved alias (tsconfig paths, through extends and references, vite/vitest resolve.alias, package.json imports) — minus the files that mock it. A test that vi.mocks / jest.mocks the target cannot detect any fault in it; counting it would make NOCOVER under-report and waste runs. NOCOVER therefore means exactly "no test file imports this source without mocking it". claims prints the split (18 import · 16 mock · 2 can detect), the evidence lists the mocking files, and a mocking file that never expect(...)s anything imported from the target carries the static signal mocked-never-asserted — the cheapest blind-spot signal there is, and the exact signature of one escaped bug in the field reports. Silence it, visibly, with // unasserted: <why> above the mock.

   Runners: vitest and jest (--runner auto picks the first that resolves; both read the same jest-compatible JSON report). A runner is resolved from the project's own package first — its pinned version, its own bin script — and only then from an executable on PATH, which the evidence records as runner.source: "path"; npx is never asked, because its cache answers for packages a project does not have. Anything else goes through --runner-cmd. Each runner is proven against its own copy of the known-answer fixture.

   The two-gate rule, as one verb: testguard admit <test-file> --claim <ID> runs the claim's faults against your uncommitted test and answers ADMITTED (exit 0: the test is green on unmodified HEAD and fails on every fault, N/N) or NOT ADMITTED (exit 1: the first blocking fault and what to do about it). The test must be a declared or discovered defender of the claim. It is sugar over probe --claim <ID> --include-dirty, so it can never disagree with the gate. Every generator on the market admits a test because it compiles, passes and raises coverage; admit admits it because it fails when the claim is false.

   Practical loop: first pass --no-escalate (escalation re-runs the whole suite N times per survivor); iterate on one claim with --claim <ID> and either --include-dirty or --in-place (only fault target files must be clean there; test files may be dirty), optionally --confirm 1 for a fast provisional signal — verdicts print with a ?, evidence goes to evidence-provisional.json, and baseline refuses it; final pass with defaults. By default the stream shows only unproven faults plus a killed count — --verbose shows every fault. A custom runner (pnpm --filter, a specific config) goes in --runner-cmd "<cmd> {files} … {out}"; if the scratch worktree cannot see your node_modules, pass --node-modules <dir>.

3. Baseline freezes every non-passing fingerprint. Later probes suppress what was already known and exit non-zero only on what is new. Claims whose source and defenders are unchanged reuse their prior verdict, so a probe in CI costs only what changed. A baseline frozen from --include-dirty evidence records the snapshot and points at the parent of the commit that will carry your tests; after you commit, a clean probe plus baseline --restamp moves it to that commit — only when the fingerprints are identical, never otherwise. status notes a baseline that predates HEAD without making it a state.

4. Brief turns evidence plus baseline into a ranked, capped ## TEST BLINDSPOT CONTEXT block, printed and also written to .testguard/brief.json (--text prints only; --markdown prints the same brief as a merge-request note for the human reviewer, unclaimed changes first). Wire it into an agent's session start — for Claude Code, in .claude/settings.json:

   { "hooks": { "SessionStart": [ { "hooks": [
     { "type": "command", "command": "node_modules/.bin/testguard brief --text 2>/dev/null || { command -v testguard >/dev/null 2>&1 && testguard brief --text 2>/dev/null; } || true" }
   ] } ] } }
   

   --text prints only, and exits 0 silently when there is no evidence yet; the command resolves the project's own install, then a testguard on PATH, and ends in true — so the hook can never break a session. There is no npx in it, in any form: --no-install resolves the package from the registry before declining to install it, so it is quiet rather than offline. The brief's first line says which install answered (local install or global), so a stale one is visible.

Would this suite have caught the bugs that already escaped?

An injected fault is a fault somebody thought of. A bug that actually shipped is ground truth: a human already confirmed it was a defect, and there is no equivalent-mutant argument to have about it.

npx testguard-cli replay --since HEAD~50..HEAD --max 10

For each fix commit in the range — one that changes source and a test together — replay checks it out in a scratch worktree, reverts only its source files to the parent, deletes the test the fix shipped (that test proves nothing about what the suite knew before it existed), and runs the tests that import the reverted code:

One patch counts once (git patch-id), because a dual-branch topology carries the same fix under two or three shas. replay reports and never gates: a bug that escaped is history, not a regression in this change.

Each replayed bug is also labelled with the injected-fault class its diff most resembles, which is the join key that turns an uninterpretable mutation score into a statement with a sample size:

  guard-removed          blind   7/9    p=0.78  ci [0.45, 0.94]
  field-dropped          blind   4/4    p=1.00  ci [0.51, 1.00]

Read as "when a fault of this class survives, how often does that correspond to a bug that really escaped". It is written as a calibration document beside the replay one. Only caught and blind carry information; the other verdicts are excluded from both sides of the ratio.

The open question this exists to answer. Does a calibration learned on a repository with history transfer to a greenfield one that has none? AI authored code has no history, so replay cannot help it directly and calibration is the only bridge. That transfer is unproven, it is the core product bet, and this is the instrument for testing it — not the answer.

Read CI's evidence locally

Verdict reuse makes a probe cheap, but the evidence lives where probe ran and is gitignored. On a fresh clone, or on a laptop where CI does the probing, the session-start brief is empty and status says unprobed while the default branch has full evidence. Point either command at CI's document:

testguard status . --evidence .testguard/ci/ci-self-evidence.json
testguard brief . --text --evidence .testguard/ci/ci-self-evidence.json

A foreign document is not trusted blindly. status marks it evidenceSource: provided, prints the commit it describes next to the commit in your tree, and still computes staleness from the recorded input hashes — so a file you have edited since CI probed it goes evidence-stale for exactly those claims.

testguard init --ci-evidence github (or gitlab) writes .testguard/fetch-ci-evidence.sh, which downloads the branch-named artifact with the platform CLI you already have and then briefs from it. It is an on-demand helper, not a hook: the session-start hook never touches the network, and the helper exits 0 with a message when the CLI or the artifact is missing. TestGuard itself still makes no network calls.

Every change needs a claim

probe can only verify claims that exist. Every escaped defect in the field reports so far was a claim gap: the feature shipped green with zero claims, and a verifier with no claim about a feature is silent about it by construction. gate closes that hole on the delta:

npx testguard-cli gate --changed origin/main            # in a PR: the files changed since the base branch
npx testguard-cli gate --changed HEAD --include-dirty   # before a commit: the working tree, staged or not

Every changed source file must carry a fault, resolve as a defender of a claim (test files), or be excused by an unexpired path entry in testguard.ignore.json — with a reason a reviewer will accept. One unclaimed file exits 1; there is no percentage. Every reliance on an ignore entry is printed, so a reviewer sees why the gate passed; an expired entry excuses nothing. Non-source files and documented never-claimed patterns (*.d.ts, *.config.*, fixtures, mocks; --explain lists them) are excluded and said so; --strict fails a change that evaluated nothing.

With a reference known, status --changed <ref> reports unclaimed-changes before any evidence state and makes the claim the next action; the brief lists the unclaimed files first. The claim is written before more code.

In CI the base is detected — GitHub Actions (GITHUB_BASE_REF) and GitLab merge request pipelines (CI_MERGE_REQUEST_DIFF_BASE_SHA, then CI_MERGE_REQUEST_TARGET_BRANCH_NAME); TESTGUARD_CHANGED_REF overrides both. The base must exist locally: GitHub — actions/checkout with fetch-depth: 0; GitLab — the diff base sha needs nothing extra on a merge request pipeline, the branch name needs GIT_DEPTH: 0 or a git fetch origin <target>. A detected base that does not resolve is a warning for status and brief (they keep working) and an error for gate (its whole job is the measurement).

# GitHub Actions
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- uses: raccioly/testguard@v0.6.0
  with: { command: gate }

# GitLab CI — the component-shaped template: gate + probe, brief as an artifact and, opted in, as a merge-request note
include:
  - remote: 'https://raw.githubusercontent.com/raccioly/testguard/v0.5.0/packaging/gitlab/testguard.gitlab-ci.yml'
    inputs: { dir: backend, post_note: true }   # post_note needs TESTGUARD_GITLAB_TOKEN (api scope); one note, updated in place

The template carries spec: inputs: (version, dir, image, severity, confirm, budget, no_escalate, strict, post_note, stage), so mirrored into a GitLab project it is a catalog component that a compliance framework can require on every project in a group. The CLI itself never talks to the network; only the job posts, and only when told to.

Properties

• Deterministic measurement. No model decides a verdict. Faults are string substitutions from a reviewed file; verdicts come from the test runner's structured report, confirmed N times. The same inputs give the same evidence, which is what makes it replayable for an auditor.
• No network, no telemetry. The CLI never opens a socket. The session-start hook resolves an installed binary and never fetches one. Nothing leaves the machine unless a CI job you configured posts it.
• Artifacts are data in the repository. Claims, evidence, baseline and brief are JSON validated against a published schema before they are written. Nothing is healed, regenerated or re-targeted at run time; a claims file is code, and is reviewed like code.
• Never optimistic. A timeout, a load failure, a mixed N-run result, a flaky defender or a missing anchor is reported as unproven, never rounded toward green. The one pass is killed; everything else gates.
• One exact-pinned runtime dependency (ajv), Node ≥ 20, MIT.

What TestGuard is not

• Not a test generator. It judges a test the agent wrote (admit); the generating half stays with the agent, where the market is putting it.
• Not a mutation-score dashboard. No blanket mutants, no single score, no threshold. Faults are few and bound to stated claims; findings are ranked, never summed.
• Not a self-healing runner. A defender that adapts itself to a change is a defender that did not observe it; TestGuard reports that, it does not do it.
• Not a coverage tool. A line executed says nothing about whether an assertion would notice. NOCOVER here means no test even imports the file; everything above that is measured by injecting the fault.

A claim that disappeared is invisible to all of the above: probe verifies the claims that exist, gate sees the source file covered by some other claim, and status says clean. Deleting a claim is therefore cheaper than weakening its fault — and changedFaults exists precisely because weakening was the cheap escape. So:

npx testguard-cli claims --since origin/main    # what existed there and does not now

removed-claim and removed-fault gate; a rename that keeps the statement verbatim is reported as renamed-claim and does not. Where evidence is present, the line names the verdict the claim last had, because "it was SURVIVED when it was removed" is the sentence that matters. The escape hatch is the same one as everywhere else: a claim entry in testguard.ignore.json with a reason, expiring if you give it an expires. Removal is allowed — claims can be wrong, superseded or split — and it is never silent.

Run it from any harness

The operating loop above lives in a Claude Code skill and a session-start hook. Both vanish the moment the harness is Cursor, Codex, Devin or whatever comes next — and the harness is exactly the layer most likely to change. So the same documents are also served over the Model Context Protocol:

npx testguard-cli mcp            # JSON-RPC 2.0 on stdio
npx testguard-cli init --mcp     # prints the config for Claude Code, Cursor and Codex

Five tools, all read-only: testguard_status, testguard_brief, testguard_claims, testguard_evidence, testguard_next_command.

No tool runs a probe. A probe is long-running, budgeted, and the person should see it happen, so next_command hands back the exact shell line for the agent to run in its own terminal. Nothing here writes a file either — editing a claims file through a connector would defeat the point of recording every fault edit. The server declares only a tools capability: no prompts, no resources, no sampling.

It is hand-written against a pinned protocol version rather than built on the official SDK, because this tool has one exact-pinned runtime dependency and keeps it that way. The surface is three methods and five tools and will not grow; tests speak the wire format to a real child process and check each tool against the CLI's own --json output, so the two cannot drift.

Built for agents to run

TestGuard is meant to be driven by an AI agent, not typed by a person. Three things make that safe:

• One source of truth. testguard status --json computes state and the one next action from the claims file, the evidence, the baseline and the working tree. Every human rendering — the CLI text, the session-start brief, the skill — derives from it, so they cannot disagree. Every command accepts --json.
• An installable operating loop. testguard init [dir] writes the agent layer at the git root, where agent sessions run — .claude/skills/testguard/SKILL.md (state → action, verdict → the only acceptable fix, the two-gate rule for any test the agent writes), the brief --text session-start hook, an AGENTS.md section — and the project layer (.gitignore lines) beside the claims file. A second project in the same repository adds a hook line and an AGENTS.md bullet; --here keeps everything in the subdirectory. The hook resolves the project's node_modules/.bin/testguard, then the git root's, then a testguard on PATH via command -v, then does nothing — no npx in any form, so no path to the network; a pre-0.6 npx -y or npx --no-install hook is replaced. A written file that .gitignore swallows is reported, not offered for commit. Idempotent.
• Independence is recorded (L3). probe measures power — would this test notice if the code were wrong. It also records, on every kill, whether the killing test was last touched by the same change (or the same author) as the code it guards: detail.independence is co-authored, separate-change or unknown. A co-authored kill is legitimate — a fix should ship with its regression test — but a repository where every kill is co-authored has no independent verification, whatever its claim verification rate says. It is a signal: ranking reads it, verdicts never do, and the brief says it in one line. Nothing else in this category measures it, and agents saturate the tests they can see (SpecBench).
• Gaming is visible. The cheapest way to make a survivor disappear is to weaken its fault, not to write a test. Evidence records every fault's content hash; status lists any fault edited after it survived, with its previous verdict, and makes reviewing that edit the next action. Editing a claim is allowed — claims can be wrong — but it is never invisible.

Authoring faults mechanically

Writing faults by hand means reading the code to find exact anchors. Two field reports found that ~80% of hand-written faults are one of nine shapes, so scaffold proposes them for you:

npx testguard-cli scaffold src/auth.ts          # → .testguard/scaffold-auth.json (a draft, never your claims file)
npx testguard-cli scaffold src/auth.ts --claim AUTH-ADMIN   # every proposal under one claim; copies it if it exists

Every proposal's find is the exact line with expectHits/occurrence computed from the file, so it is verifiable by construction; provenance is producer: derived; defendedBy is prefilled from the tests that import the module; proposals are grouped under a preceding @claim <ID> annotation or by enclosing function. Statements are TODO: placeholders — a proposal becomes a claim only when a human states what it defends. Deterministic heuristics, no AST, no LLM; a proposal the tool cannot anchor is never emitted.

Commit .testguard/baseline.json; ignore evidence.json and brief.json. The baseline is the frozen contract; the other two are regenerated per run.

The fault model is the auditable artifact. You never reach 100% of correctness; you reach 100% of stated claims verified, and the statement of claims is what an assessor reads. A claims file is code — its replace strings run under your test runner — so review it like code.

Try it

The repository ships a known-answer fixture with a real blind spot:

git clone <this repo> && cd testguard && npm install
npm test                                  # includes probing the fixture end to end

fixtures/known-answer/ is a tiny project whose audit-row test asserts with expect.objectContaining({...}) and omits the content key. Swap the redacted text for the raw input and the test stays green. probe reports it as SURVIVED; the fixture's README walks through every verdict.

Status

v0.5. Eleven commands (status, init, claims, probe, admit, replay, baseline, brief, gate, scaffold, mcp), vitest and jest runners, hand-authored faults plus v0.5. Nine commands (status, init, claims, probe, admit, baseline, brief, gate, scaffold), vitest, jest and Playwright runners, hand-authored faults plus a mechanical scaffold, an agent operating layer (status, init) and a change gate (gate). The contract spine — eight JSON Schemas shared with the other Guard tools — is under spec/. One exact-pinned runtime dependency (ajv, for schema validation); Node ≥ 20.

Not yet: test generation (the acceptance half, admit, exists; the generating half stays the agent's), runners beyond vitest and jest, AST-aware producers, and the transfer of a calibration between repositories — replay measures it now; whether it carries to a repository with no history is unproven. Each is designed for; none is claimed. vitest, jest and Playwright, AST-aware producers, and calibration of fault classes against real escaped bugs. Each is designed for; none is claimed.

Licence

MIT.