Countercheck

Proof that AI work really happened.

AI coding agents often say "done, all tests pass" when it isn't true: they skip a failing test, edit the test so it passes, or never run it. countercheck verify checks the claim itself:

1. It reads your checks from the base revision (before the agent's change), so the agent can't weaken them.
2. It copies the agent's files into a clean folder and puts back any protected file the agent changed (your tests).
3. It runs the checks in a macOS sandbox: no network, writes only inside that folder.
4. It writes a receipt: the claim, what ran, the output, what the agent changed that it shouldn't have, the cost if known, and a verdict. Receipts are sealed with a SHA-256 hash, so edits are detectable.

$ countercheck verify --claim "Fixed the discount bug. All tests pass." --agent claude-code --cost-usd 0.42
countercheck receipt 9073a384e88f9d37  FAILED
  claim:    "Fixed the discount bug. All tests pass." -> CONTRADICTED by evidence
  checks:   .countercheck/checks.toml @ HEAD (locked), sandbox: macos-seatbelt (no network, writes only in the check folder)
  protected Modified: tests/test_pricing.py -> restored_from_base
  [FAIL] unit tests  (65 ms)  $ python3 -m unittest -q
         |     self.assertEqual(apply_discount(10, 150), 0)
         | AssertionError: -5.0 != 0
  cost:     $0.4200 (reported)
  - the worker changed 1 protected file(s); countercheck ran the checks with the base version
  - check "unit tests" failed (exit 1)

(The agent's own run said OK (skipped=1): it had marked the failing test as skipped.)

Install

Builds: macOS (Apple silicon and Intel, one universal binary) and Linux (x86_64, arm64; static). The install script checks every download against SHA256SUMS.txt before installing. Windows is planned; use WSL for now.

Sandbox by platform. On macOS, checks run in the system sandbox (no network, writes only in the check folder). On Linux there is no sandbox yet: checks run as a normal process and the receipt says so; run Countercheck inside a container with networking off when you need isolation.

What's shipped, in progress and next: docs/ROADMAP.md.

Quick start

countercheck init                      # writes .countercheck/checks.toml, edit the commands
git add .countercheck && git commit -m "Countercheck checks"
# ... let an agent work ...
countercheck verify --claim "what the agent said"            # uncommitted work vs HEAD
countercheck verify --base main                              # committed work on a branch vs main
countercheck show .countercheck/receipts/<id>.json                   # re-check a receipt's seal

.countercheck/checks.toml:

protected = ["tests/**"]          # the agent may not change these; .countercheck/** is always protected
link = ["node_modules", ".venv"]  # git-ignored dependency folders to make visible (read-only)

[[check]]
name = "unit tests"
run = "pytest -q"                 # any command; runs with `sh -c` in the clean copy
timeout_secs = 600

Besides plain commands there are matrix, forbid, golden, budget and coverage checks: see docs/checks.md and the runnable example in examples/data-migration.

Exit codes: 0 verified · 2 failed (claim contradicted) · 3 unverified (nothing failed, but the checks weren't locked in the base revision, or none ran) · 1 error · 4 (countercheck show) receipt was edited.

Agent hooks

• Hermes Agent: countercheck hook hermes-pre-verify keeps Hermes working while the checks fail.
• Claude Code and Cortex Code: countercheck hook claude-stop blocks the Stop while the checks fail. See docs/hooks/claude-code.md.

Status

Early (v0.1, macOS first). Not yet: signed receipts (Ed25519), cost read from agent logs, read-back checks against real systems (databases, CRMs, web pages), a receipts UI, and Linux/Windows sandboxes. See docs/decisions/.

Build

Rust 1.90+: cargo build --release → target/release/countercheck. Gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test. Countercheck also verifies itself: countercheck verify (checks in .countercheck/checks.toml).

Repository

• crates/countercheck-core: pure logic (check file, protected paths, receipts, verdicts)
• apps/countercheck: the CLI (git, clean copy, sandbox, runner)
• eval/, docs/eval/: the evaluation against Hermes Agent (tasks with hidden tests)
• packaging/, install.sh, pyproject.toml: release packaging (Homebrew, install script, Python wheels)
• docs/decisions/: design decisions

MIT licensed.