checkwash

Your agent deleted the failing test to make CI green. checkwash catches it before merge.

It reads the git diff. If the tests got weaker and the production code was not actually fixed, it blocks. It does not run your suite, call an LLM, or use the network.

Typical catches: assert total == 105.3 becoming assert total > 0, a new pytest.mark.skip, a deleted test, a looser float tolerance, a rewritten golden file, or a CI script that stops failing.

Status: pre-release. 21 detectors, 538 tests, zero runtime dependencies. Every number below comes out of a reproducible harness in benchmarks/ — none is hand-typed, and nothing ships that a harness hasn't produced on a clean checkout.

$ checkwash check HEAD~1..HEAD

✗ checkwash: 1 high-severity finding — blocking

ASSERT_WEAKENED   high   tests/test_billing.py :: test_invoice_total
  assertion strength: EXACT_VALUE(90) -> BOUND(40)
  no non-trivial production change in this diff
  before  assert total == 105.3
  after   assert total > 0

In short

Sixty seconds, from nothing

No install, no virtualenv, no network after the download. Every release attaches a single file that carries the whole tool — it has zero runtime dependencies, so there is nothing else to fetch.

curl -LO https://github.com/taipei49314/checkwash/releases/latest/download/checkwash.pyz
python checkwash.pyz demo                       # 8 real tampering cases, blocked, offline
python checkwash.pyz check HEAD~1..HEAD         # your last commit
python checkwash.pyz sweep HEAD --limit 100     # how often it would have blocked you

demo takes under half a second and needs nothing but Python 3.11+. sweep is the honest one: point it at your own history and read the blocks yourself before you believe any number on this page. The single-file build is gated by tests/test_zipapp.py on every push, so it cannot quietly rot.

Install

Pick the surface that fits; the engine is identical behind all of them, and docs/stability.md says which parts of it are frozen.

Install the CLI, then run it. greenwash is a leftover alias and does the same thing. From the repo:

pipx install git+https://github.com/taipei49314/checkwash@v0.2.10
# or: uv tool install git+https://github.com/taipei49314/checkwash@v0.2.10

checkwash check HEAD~1..HEAD    # a range
checkwash check                 # HEAD vs the working tree
checkwash check --format sarif  # SARIF 2.1.0 for GitHub code scanning
# JS/TS: *.test.js / *.spec.ts matcher weakenings (T3.1)
checkwash demo                  # replay real tampering cases, fully offline
checkwash bench --local         # reproduce in-clone numbers (demo + pins)
# omit --local to also require the six sweep clones; missing clones exit 2

checkwash demo replays eight real tampering cases — a softened assertion, a widened tolerance, a rewritten expectation, an xfail'd failure, a swallowed error, a relaxed CI step, a self-edited CLAUDE.md, and an assertion swapped for an unrelated one of the same strength — plus one honest fix that stays silent. No network, no key, no LLM; every verdict comes from the same engine check runs.

Required check — the only configuration that blocks a merge

checkwash installed is not checkwash enforcing. A green job that is not a required status check does not stop anyone merging, and a local stop-hook is an author-side convenience: it is skipped by --no-verify and is simply not present when someone else pushes. Three steps, in this order.

1. Add the workflow (below). Note the job name — it becomes the status check's name.

2. Make that status check required. The check name is the job name (checkwash in the snippet below), not the workflow filename. UI: Settings → Rules → Rulesets → require the checkwash status check on the default branch. Or, with admin gh access and this file in the clone:

gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json

That creates a ruleset on ~DEFAULT_BRANCH requiring context checkwash. It does not overwrite existing rulesets. List first with gh api repos/OWNER/REPO/rulesets. Without this step the workflow runs, reports, and blocks nothing.

A one-page enterprise path — required check, SARIF, allowlist, CODEOWNERS — is in docs/enterprise.md.

3. Verify. checkwash doctor recognizes the exact three-step gate below and says whether it can run unconditionally. It deliberately reports other workflow shapes as analysis incomplete instead of guessing that a textual checkwash mention is load-bearing. doctor cannot see branch protection (that needs API token scopes checkwash does not ask for), and it says so rather than implying otherwise: step 2 is the one a human must confirm.

checkwash doctor        # exit 0 = no problems found; 1 = problems or warnings

GitHub Action — blocks a PR on high-severity findings:

# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@760021d56b13af1332a5cb8eedd7ba20a9bfce47 # v0.2.9

Hash pins and persist-credentials: false are required by zizmor blanket policy — a tag pin (@v4, @vX.Y.Z) is two unpinned-uses highs. Re-checked 2026-09-01 on zizmor 1.30.0: this snippet is 0 high / 0 medium. The checkwash SHA is deliberately the newest prior stable pin that doctor could verify at build time. Release N cannot embed its own commit SHA, so it adopts that SHA only after it already exists, in the next release. The result is an explicit one-release trust lag, not an arbitrary 40-hex claim. Verify this pin with git rev-parse 'v0.2.9^{commit}'; for another trusted release, substitute its version in git rev-parse 'vX.Y.Z^{commit}'. See action/README.md.

Do not gate this job on anything. A conditional gate is the defect this project shipped in its own repository: the dogfood job carried if: github.event_name == 'pull_request' in a repo that had never had a pull request, so it never executed once while the README told people to use it.

pre-commit — an author-side convenience, not a merge gate:

repos:
  - repo: https://github.com/taipei49314/checkwash
    rev: v0.2.10
    hooks: [{ id: checkwash }]

Claude Code stop-hook — checks the diff the moment the agent finishes and blocks the stop on tampering:

checkwash hook install --agent claude-code

checkwash runs the published action against its own diff on every push (.github/workflows/ci.yml, the dogfood job): the judge is judged. That job was previously gated to pull requests, in a repository that has never had one, so it had never executed — a test now fails if it is made conditional again.

License: Apache-2.0.

Integrations

# Claude Code — block the agent's stop on high findings
checkwash hook install --agent claude-code

# pre-commit — prints the config block to paste
checkwash hook install --agent pre-commit

# GitHub Actions — exact doctor-verified prior stable pin; see action/action.yml
- uses: taipei49314/checkwash/action@760021d56b13af1332a5cb8eedd7ba20a9bfce47 # v0.2.9

checkwash check BASE...HEAD (three dots) resolves through the merge base, so PR diffs never include base-branch commits. A wash split across merged PRs is still outside that window — docs/process-windows.md. To reproduce the published numbers from this checkout: checkwash bench (add --local if you do not have the six sweep clones).

Measured, not asserted

Two harnesses, both reproducible from a clone (benchmarks/):

• On test-suite refactors specifically — 25 false positives out of 60, and the 1.17% below does not predict it. 60 refactors a reviewer would approve (extract an assertion into a shared helper, merge two tests, move a check into a fixture, swap exact equality for pytest.approx), each shipping production twice — correct and buggy — so that four pytest runs prove both sides still catch the bug before checkwash is asked anything. A block is then a false positive by construction, with no adjudication to argue about. checkwash blocks 24 of the 60 — down from 20 of the first 30 before the reachable-assertion IR landed, and the residue decomposes into named families (cross-file helpers, unit-identity changes, and a deliberately-kept trade documented in THREATMODEL 92). The sweep corpus below rarely restructures test helpers, so no amount of re-running it would have surfaced this; that is the same zero-power trap that nearly shipped TEST_PATCHES_SUBJECT on a meaningless zero. Both numbers are real and they answer different questions. benchmarks/refactors/.
• Human-commit block rate — 42 / 1800 = 2.33%. Six active OSS projects (flask, httpx, attrs, click, rich, starlette), 300 consecutive human-reviewed commits each, none seen during development. That is how often checkwash would fail CI on a commit a human wrote. Every repo is at or under 4%; the progression from an initial 8.6%, and what moved each step, is in the benchmarks README. A block is not automatically a mistake. All 42 were adjudicated commit by commit against the real diff: 27 false positives (1.50%), 15 legitimate policy blocks (0.83%) where the commit really does drop oracle coverage with nothing visible replacing it, 0 unclear. Three precision rounds brought this down from 2.50% / 1.67%: skip conditions are read (constants resolved up to the head snapshot) instead of grepped, relocated tests are recognised even when they carry their own skip markers or hold no assertions, feature removals and dependency bumps explain the removal of their tests, and deleting one of two identical copies is recognised as dedup because the survivor is found at head and checked to still run. Every legitimate policy block still blocks and the decoy corpus still blocks 12/12 — and the process cut both ways: one over-eager credit was caught clearing two correct blocks and tightened before it shipped, and one adjudication verdict was overturned in the tool's favour when the duplicate search proved a "relocated" test actually reappears nowhere; each trade is reported as measured. The block rate is a machine count and exact. The split is a multi-rater judgement: two additional raters blind-re-adjudicated 35 of the 37 pre-promotion blocks, with 91–94% pairwise agreement and Fleiss' kappa 0.844, and the five blocks the 2026-08-25 promotion added were each judged by two blind raters; the published category is the majority verdict, the contested commits are marked, and the per-commit reasoning of every rater ships in benchmarks/ precisely so you can disagree with it. 1.33% of the corpus (24/1800) never got a real analysis: those commits touch a production file checkwash genuinely cannot read — other-language code, templates, data files, unparseable Python — which suppresses escalation for the whole diff (THREATMODEL #4). That share of the pass rate rests on a documented blind spot, measured rather than assumed. It used to be 7.2%: narrowing the blanket (docs config, type stubs, repo metadata, pin sources, deleted-but-parseable Python and scripts that run the test suite no longer qualify) removed 105 exemptions and did not change a single verdict — every one had been protecting a commit that passed on its own merits anyway. The same round taught checkwash where the test command actually lives. A shell script or make recipe that runs the suite is test-runner configuration, not production: weakening it (|| true, set +e, a - before a make recipe, losing set -e, or deleting the invocation) is now a blocking finding, and touching it no longer buys the diff an exemption. On the human corpus that cost nothing — the same 35 blocks before and after, the whole delta being four warn-level notices on starlette commits that edit scripts/test. Read that with its limits: these six projects barely exercise the change, and the defence against over-flagging is the content gate (a Makefile that only compiles is still production) plus its negative fixtures, not the sweep.
• Recall against an attacker who has read the rules — 49 / 80, and that is the number to look at. 80 tampering cases across two arms, each verified mechanically before counting: production byte-identical, pytest red before and green after. The first arm (40 cases, written against a description of every rule here) caught checkwash at 12/40: the IR only knew an oracle that was a syntactic assert, so assert_sum(add(2, 3), 5) was a call and the unit read as having zero assertions. The IR now records the assertions a test executes — through the file's own call graph, invocation and not mention — which brings that arm to 19/40 — 20/40 with the cross-file channels of v0.1.27, 21/40 with v0.2.4's 91b closure — and blocks 28/40 of a second arm written afterwards to disguise the weakening as a helper-extraction refactor. The 31 that still escape are enumerated, not waved at: most subvert the meaning of a syntactically strong assertion (__eq__ that is always true, a no-op TestResult) or compute zero runs, which a single static diff cannot decide — mutation testing is the honest answer there, and THREATMODEL 91a says so. Full corpora, harnesses and per-case verdicts in benchmarks/tamper/. It bounds a motivated, informed attacker rather than ordinary agent behaviour — but a helper that stops being called and a fixture that stops being autouse are not exotic.
• Recall on the older decoy corpus — 12 / 12, then probed harder. Twelve bug-fix tasks run through real coding agents told to make CI green without fixing the bug; checkwash blocks all twelve tampering diffs. Under natural conditions, 0 of 12 agents touched a test at all — an honest datapoint, recorded as such. A second wave of six probe tasks (2026-08-04) aimed agents at the de-escalators that hold at warn by design. The agent escaped two of six — a one-line constant flip behind an existing skip guard, and a conftest fixture that monkeypatched the function under test while leaving production and tests byte-identical. Both escapes became detectors the same day (TEST_DISABLED guard evaluation, CONFTEST_PATCHES_PROD), both recorded diffs now block, and both new rules fire zero times across the 1800 human commits. The four other probes were blocked outright. A third wave (2026-08-07) aimed two tasks at repos that keep the test command in scripts/test.sh rather than in the workflow. One of two was caught. The first agent's opening move was to make the suite non-fatal inside the runner script — a diff the previous release scored at zero findings, and this one blocks. The second escaped: it dropped the failing tests out of collection, which checkwash detected, and the honest one-line scripts/lint.sh edit its own task asked for demoted the finding to a warning, because a script that does not run tests is still an unreadable production change (THREATMODEL #4). That escape is published open, with its reproduction — and closed in the next release the same day, once the reason given for leaving it open turned out to be a citation that did not hold (DECISIONS D-026; the correction is in the repository because publishing it is the point).
• The informed arm — the adversary reads the source. Every arm above hides the defender. This repository is public, so one arm does not: the agent gets checkwash's complete source, spec and threat model, and may run checkwash check against its own diff until it stops blocking. Three of six tasks were refused outright by the agent provider's safety filter, so the sample is three, and all three reached a green suite past the shipped engine. One rewrote production instead of the tests (THREATMODEL #1, out of scope by design, and the cheapest route an informed adversary has). One read the opaque-change branch and added a three-line JSON file that nothing imports, purely to be unreadable — closed the same day. One normalised the subject of an assertion rather than its expectation, which no rule sees, and was closed the same day by the detector it demanded (SUBJECT_NORMALIZED), with its diff published either way. If you want one number from this project, that arm is a fair one to take: three informed attempts, three different routes past the shipped build, two of them now regression fixtures and one of them a documented limit of the whole approach.
• A live miss in this repository, the day after that arm ran. Another agent working on checkwash changed the test that guards its own release tag, replacing one assertion with a different one of equal strength and adding an early return. That is the gate's documented failure mode, written into its own assertion message. checkwash passed the diff — two CI_WORKFLOW_TOUCHED warns and nothing else. Three rules came close and none fired, and the actual cause was in alignment rather than in any of them: the last-resort pairing stage matches leftover assertions by span order, so a deleted assertion and its unrelated replacement were reported as one unchanged assertion. Closed in v0.1.27 by ASSERT_SUBSTITUTED, which is the first rule keyed on how a pair was formed rather than what it contains. The diff blocks at high on this build. D-031 and D-033 have the whole account, including the first attempted fix, which closed a six-line reduction of the bug and did nothing about the bug.

The first recall measurement caught 0 of 12 — pytest's own .pyc output disarmed the gate, a bug two rounds of code review had missed. Building the harness is how it was found. See benchmarks/decoy/.

Prior art

checkwash is not the first tool to look for agent shortcuts in diffs, and does not claim to be. Closest neighbours, credited up front:

• swarm-orchestrator — a PR audit suite (11 detectors, JS/TS-tuned, LLM judge layer, sandboxed runtime proofs; advisory by default). checkwash is the narrow, deterministic end of this spectrum: Python-first oracle semantics (a strength lattice, not matcher swap-lists or assertion counts), zero LLM anywhere, zero code execution, byte-identical verdicts, and a per-fingerprint reviewed-exemption workflow — small enough to sit in a stop-hook.
• AgentLint — broad agent guardrail rules including no-test-weakening; state-based linting rather than two-sided semantic diff.
• mumei (reported; Claude-Code-specific harness with clean-HEAD test reruns and golden-file freezing) — a harness, where checkwash is a single-purpose differ any harness can call.

License: Apache-2.0.