checkwash

Check whether a code change weakens your tests.

A test can pass after it stops checking something useful. checkwash reads your Git diff and flags known patterns such as deleted assertions, skipped tests, relaxed expectations, and CI checks that stop failing.

- assert total == 105.3
+ assert total > 0

The new assertion accepts many incorrect totals. checkwash can flag changes like this for review, including changes written by coding agents.

Runs locally. No LLM. No network during analysis. Never executes your code.

v0.3.0 replaces five file-wide exemption rules with content-bound ones. In v0.2.13 those rules could reuse a recorded approval for a later change to the same path; v0.3.0 retires their path-only keys. The upgrade notes describe the content-bound replacement, retired keys, and installation fixes.

v0.3.3 fixes two measured false positives involving shared immutable fixture parameters. It retains the existing execution-context and bounded-oracle checks. Release evidence and remaining limits

Try it

You need Python 3.11+ and Git. Download and try the offline examples first.

Windows PowerShell (including 5.1):

curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.3.3/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

PowerShell 5.1 aliases curl to Invoke-WebRequest; use curl.exe as written. macOS/Linux or Git Bash:

curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.3.3/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

Then, from a repository with at least two commits:

python checkwash.pyz check HEAD~1..HEAD

This checks your last commit. Start with a change you already understand. You can also download the file in your browser. For uncommitted changes use python checkwash.pyz check. For a branch review, use python checkwash.pyz check BASE...HEAD to compare from the merge base; BASE..HEAD compares the two named snapshots directly.

The default threshold is high: a visible warn can still pass. REPAIR_EVIDENCE describes related changes in the same diff; it does not prove a repair is correct.

For JSON/SARIF output and more examples, see the usage guide.

Know the limits

v0.3.3 is alpha. A pass does not prove that a change is correct or honest. Python is the main language supported; JS/TS support covers a limited set of test patterns. Known gaps remain.

Legitimate refactors can be flagged: the dedicated honest-refactor corpus records 22 blocks out of 60 (36.7%). The stricter runtime verifier qualifies 57 cases; three have existing fixture errors, and 20 of the qualified cases block. Try it on your own changes before making it required. Coverage and limitations · Known gaps

Use it in CI

To stop a merge, make the checkwash status check required in your repository's branch rules. Installing the tool or adding a workflow alone does not enforce its verdict.

The recommended Action is pinned to v0.3.2; the CLI above is v0.3.3. The shared-fixture precision fix below is in the CLI; the prior-release Action does not include it. Record which version you use. Full setup and exemptions

# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@4ad6e31019dd40e69f46f3f0ac24d37bbdad31ac # v0.3.2

gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json

More options and evidence

pipx install checkwash==0.3.3
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.3.3

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline

Related projects: checkwash-corpus and smallestlie hold evaluation work.

Alpha pre-release. 21 detectors, 1484 tests in the current source tree. Zero runtime dependencies. Apache-2.0.