checkwash

v0.8.0's Action no longer runs code from the pull request it judges; commands run from a subdirectory read the repository from its root; and oversized files that no detector reads are carried unread instead of stopping the run. Release evidence and known limits. A pass does not prove correctness; retain exit 2 and diagnostics as analysis errors. With --format hook-json (the Claude Code Stop hook) an engine error now exits 0 with a "not analysed" message. The recommended Action is pinned to the same v0.8.0 release.

Development snapshots: the source in this checkout may be newer than the published CLI. The advertised-release manifest records the version, commit and download bytes behind the installation commands below. New changes require their own candidate evidence and remain unreleased until the publication record verifies them; candidate CI success does not mean that the recommended Action pin includes them.

Check whether a code change weakens your tests.

A test can pass after it stops checking something useful. checkwash reads your Git diff and flags known patterns such as deleted assertions, skipped tests, relaxed expectations, and CI checks that stop failing.

- assert total == 105.3
+ assert total > 0

The new assertion accepts many incorrect totals. checkwash can flag changes like this for review, including changes written by coding agents.

Runs locally. No LLM. No network during analysis. Never executes your code.

v0.3.0 replaces five file-wide exemption rules with content-bound ones. In v0.2.13 those rules could reuse a recorded approval for a later change to the same path; v0.3.0 retires their path-only keys. The upgrade notes describe the content-bound replacement, retired keys, and installation fixes.

v0.4.0 extends detection of runtime suppression, collection changes, expectation rewrites and substituted subjects, and recognizes more table refactors. The proofs retain input identity, execution context and conservative handling of unknown code. Release evidence and remaining limits

v0.4.1 repairs Node assertion recognition (#164) and covers Node test-file names, bounded import aliases and lexical shadows. Coverage warnings expose known assertion candidates the scanner cannot represent; they do not prove complete JS/TS coverage. The same assertion contract checks the source, wheel and zipapp. v0.4.1 evidence and limits

v0.4.2 strengthens JS/TS assertion evidence. Test callbacks own their assertions; changing a scalar expected value or loosening positive toBeCloseTo precision reaches the existing detectors. Equivalent scalar spellings remain equivalent across subject checks. This is bounded syntax support, not general JS/TS analysis. v0.4.2 evidence and limits

v0.5.0 closes a round of red-team reports (#172–#181). A skip in a fixture or setup the test runs, a selector over edited tests, a package manifest's test command, Jest's __tests__/ layout, JS block-level and imperative skips, JS module mocks, hand-rolled JS tolerances, chai assertions and CI control flow that stops a runner now reach the detectors; #173, #180 and #181 are closed in part. Some existing findings now carry a different fingerprint, so allowlist entries for those shapes must be re-recorded; the JSON shape is unchanged. v0.5.0 evidence and limits

v0.6.0 works through the reports and rulings that followed (#196–#261). JS/TS assertions keep their meaning across Jest, chai, Vitest and node:test, and the runner a file's imports name decides whether a moved test still runs and what a focus stops. Hand-rolled tolerances in both frontends, pytest selectors under explicit targets, skip guards in a body, a setup or a conftest hook, every pytestmark spelling and a CI config the reader cannot read now reach the detectors. Some existing findings now carry a different fingerprint, so allowlist entries for those shapes must be re-recorded; the JSON shape is unchanged. v0.6.0 evidence and limits

The next minor (unreleased) makes assertion-level exemptions content-bound (#421). Through v0.8.0, an exemption for one rewrite of an assertion also allowed every later rewrite of the same line. Nine assertion-level rules now key a finding by the old and the new assertion as well, TEST_DISABLED keys one by what the change disabled (#512), and their old keys stop matching: re-review and re-record those entries. Assertion-level notes

Try it

You need Python 3.11+ and Git. Download and try the offline examples first.

Windows PowerShell (including 5.1):

curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.8.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

PowerShell 5.1 aliases curl to Invoke-WebRequest; use curl.exe as written. macOS/Linux or Git Bash:

curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.8.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo

Then, from the root of a repository with at least two commits:

python checkwash.pyz check HEAD~1..HEAD

This checks your last commit. Start with a change you already understand. You can also download the file in your browser. For uncommitted changes use python checkwash.pyz check. For a branch review, use python checkwash.pyz check BASE...HEAD to compare from the merge base; BASE..HEAD compares the two named snapshots directly.

From a subdirectory, check analyses the whole repository from its Git root and says so in one line on stderr (since v0.8.0, #416). Before v0.8.0, run it from the repository root.

The default threshold is high: a visible warn can still pass. REPAIR_EVIDENCE describes related changes in the same diff; it does not prove a repair is correct.

For JSON/SARIF output and more examples, see the usage guide.

Know the limits

v0.8.0 is alpha. A pass does not prove that a change is correct or honest. Python is the main language supported; JS/TS support covers a limited set of test patterns. Known gaps remain.

In v0.8.0, JS/TS test.each, it.each and test.for tables are not scanned: an assertion weakened or deleted in one, an edited row or a deleted table passes. From the next minor (unreleased), each such table is a test unit, and its rows are read when they can be counted (an inline array or a tagged template, or a same-file const read only as a table). The rows of a describe.each table stay unread, and a table with a computed title is not a unit (#428).

Legitimate refactors can be flagged. The frozen historical corpus records 22 blocks out of 60 (36.7%). A separate pre-release source replay records 4/60 blocks, including 2/57 strictly qualified cases; three existing fixture errors remain. These are selected examples, not a general false-positive rate.

A changed file over 1,000,000 bytes is not read. An image, data file, lockfile, document or non-Python production source that large is reported on stderr and earns no repair evidence. From the next minor (unreleased, #454), --format json also lists it under unread, SARIF names it in a tool execution notification, and the term report adds a ! line and, when there is no finding, opens with "analysis incomplete"; the Stop hook's JSON and --emit-ir do not name it. A file a detector must read (test, conftest, CI, guardrail, snapshot, JS setup, runner script or test-command file) that large stops the run with exit 2, naming it. So do a .py module that large, whether added, deleted, renamed or edited, and a pytest configuration file (pytest.ini, .pytest.ini, pytest.toml, .pytest.toml, tox.ini, setup.cfg, pyproject.toml) at any depth: either can change the module a test's import resolves to. So do any non-generated file under the root scripts/ directory, read for the pytest commands it runs, and any other changed file that large the analysis needs as context. A shell shebang gives a file the shape of a runner script, whatever its name, except a .py module or a JS/TS test file. From the next minor (unreleased, #455), checkwash reads the first 200 bytes of a side over the limit to tell, and uses them for nothing else; a Git side is streamed and checked against its object id. Where those bytes are read, a file that large whose side starts with a shell shebang stops the run, naming that side, whatever its suffix (a .py module or a JS/TS test file that large stops the run anyway, by its own rule). A file with no extension (a dotfile such as .envrc counts) without one, such as a README or a LICENSE, is set aside like any document, unless another rule above makes it one a detector must read. v0.8.0 reads no byte of such a side, so every file with no extension that large stops the run there. On main a Git side over 64,000,000 bytes is not streamed, so its first bytes are not known: a file with no extension still stops the run when one of its Git sides is that large, and a suffixed file whose shebang is only on such a side is still set aside. When a file that large stops the run, from the next minor (unreleased) --format json prints {"unread": [...]} on stdout instead of nothing: one unread entry, for the side the run refused. It is not a findings document, and term and SARIF output stay empty. With a range, json output stays empty too when the refused file is a symlinked pytest configuration's target that no detector must read. Generated output (build/, dist/, node_modules/ and the like) is set aside like a production source unless it is a Python module, a pytest configuration, a CI file, a runner script or a file with no extension whose shebang is not known. A root poetry.lock or uv.lock that large at the base is skipped with a note on stderr, and the skip is not listed under unread; when the diff also changes that lockfile, the change's sides over the limit are listed like any other. A root pyproject.toml or requirements*.txt that large at the base stops every run, naming it: term, json and SARIF output exit 2 and name it on stderr only. The Claude Code Stop hook (--format hook-json) reports an engine error, this one included, as a non-blocking "not analysed" message on stdout that quotes the error, and exits 0. From the next minor (unreleased), when the run stops on a file the hook's diff changes and that file's working-tree side is over the limit, the hook blocks the stop instead, naming the file and asking to shrink, split or revert it (#456). A file whose base side alone is over the limit, a root manifest over it at the base, and a padded file committed before the stop keep the message; the full list is in subject integrity.

Try it on your own changes before making it required. Coverage and limitations · Known gaps

Use it in CI

To stop a merge, make the checkwash status check required in your repository's branch rules. Installing the tool or adding a workflow alone does not enforce its verdict.

The recommended Action and the CLI above are both v0.8.0. Record which version you use. Full setup and exemptions

Advisory: Action trust boundary. Every published Action release up to and including v0.7.0 runs python -m pip and then checkwash with the pull request's checkout as the working directory, so a module in that checkout (for example a top-level pip package) can replace the installer and then the checker, and the pull request can tamper with the result it is judged by. Under checkwash's threat model that applies to every pull request it judges. v0.8.0 fixes it (#372); pin the v0.8.0 Action below. If you stay on v0.7.0 or earlier, treat a pass as advisory, review the pull request before merging and keep comment-pr off. Known issue

# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@f02b5d8a07fa39ee1898881f66c571eb164cb11c # v0.8.0

gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json

More options and evidence

pipx install checkwash==0.8.0
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.8.0

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline

Related projects: checkwash-corpus and smallestlie hold evaluation work.

The package also includes a limited quality preview for coverage, Ruff and mypy configuration review. Setup and CI adoption guide. Its frozen legacy-byte comparison remains red; publication does not establish natural-case acceptance or effective enforcement.

Alpha pre-release. 22 detectors, 20488 tests in the current source tree. Zero runtime dependencies. Apache-2.0.