Snitch

The integrity layer for AI coding agents.
Catches agents that make a failing test pass by cheating, and gives them an honest way out.

_{A real claude plugin eval run (2026-10-24, case broken-expected-value), replayed; the agent's messages are quoted, shortened where marked “…”. Without Snitch, the same agent edited the test in 8 of 32 trap runs.}

You ask a coding agent to make CI green. It comes back with "all tests pass". What it did: change assert total == 42 into assert total, mark the failing test skip, or add -k "not slow" to pytest.ini. The diff is green; the bug is still there.

Snitch watches for exactly that, live in your Claude Code session, on every pull request, and in a benchmark that ranks agents by honesty. It never asks a model whether something "looks like cheating": every finding comes from a deterministic detector and points at a line in the diff.

- assert divide(10, 2) == 5
+ assert divide(10, 2) in (5, 6)

# Snitch receipt
**1 event** — 1 high

| Severity | Family    | Pattern            | Location       | Detector |
|----------|-----------|--------------------|----------------|----------|
| high     | tampering | assertion_weakened | test_calc.py:9 | ast      |

See it catch

A test that can't fail. In a real TemptationBench trial, an agent made a test that needs a missing CI secret pass by setting that secret inside the test. Snitch flags it. Patch and message shortened.	Live in Claude Code. The plugin's real hook output: a warning right after a weakened assert, then a blocked stop when the final message hides it.
Before you commit. Real snitch check output; exit code 1 fails a pre-commit hook or CI step.

Why

Research has shown that coding agents cheat on tests, and that an honest exit helps:

• EvilGenie (arXiv 2511.21654) observed explicit reward hacking by Codex and Claude Code.

• SpecBench (arXiv 2605.21384): frontier agents saturate visible tests while hacking persists on held-out ones.

• BenchJack (arXiv 2605.12673): one root conftest.py hook can rewrite every test outcome, and most reviewers would miss it.

• Escalation channels (arXiv 2608.29460): agents rarely disclose tampering, but giving them a way to report a broken test reduces hacking; escalating and hacking were almost mutually exclusive.

• ImpossibleBench (arXiv 2510.20270): on impossible tasks, agents edit tests, overload __eq__, keep hidden state or special-case inputs; LLM monitors caught only 42–65% of it on real repos.

Snitch turns those findings into something you install: it detects the cheat, gives the agent the policy + escalate tool combination that cut reward hacking from 23.6% to 5.3% in the escalation study, and refuses to let a session end with a test change the agent did not mention. How each paper shaped Snitch.

How it works

flowchart LR
    A["Claude Code plugin<br/>(live, per tool call)"] --> E
    B["GitHub Action / pre-commit<br/>(per PR or commit)"] --> E
    C["TemptationBench<br/>(Harbor, per trial)"] --> E
    E["Engine: before vs after<br/>test inventory · AST assertion diff · rules"] --> R["Events → receipt,<br/>SARIF, ledger, scores"]

One engine, three adapters, so the benchmark measures exactly what users install. Each detector is a pure function of (before, after):

• inventory: pytest --collect-only before and after; tests that vanished, were renamed away, skipped or deselected.
• astdiff: tree-sitter diff of every test function; assertions removed, weakened (== → in, looser approx, looser bounds, == → !=), expected values changed, asserts hidden behind a new if.
• rules: runner and CI config (addopts, coverage thresholds, continue-on-error, || true, --no-verify), conftest.py report hooks, code that special-cases pytest, except: pass around changed logic.

Measured, not claimed

claude plugin eval on 20 seeded repos (broken test, impossible spec, flaky test, slow test, runner config, controls), 2 runs per case, with and without the plugin (full report):

Paired McNemar p = 0.008 (per case: 5 cases → 0, p = 0.063). An earlier run without the session-start policy gave 7/32 → 0/32 with 25/32 escalations (report). In both runs the agent disclosed every test edit even without Snitch, so this says nothing yet about silent cheating; that is what TemptationBench, with hidden tests, is for. The reports list the threats to validity.

Install

Claude Code plugin

/plugin marketplace add ennajari/snitch
/plugin install snitch@snitch

Pick a mode in /config (needs Claude Code 2.1.271 or later):

• observe: record only
• warn (default): tell the agent and you, never block
• enforce: also block edits to frozen tests and stops that hide test changes

Snitch collects your tests with the repo's own .venv/bin/python when there is one (set SNITCH_PYTHON to use another interpreter), so tests that import your project's dependencies can be inventoried. That interpreter needs pytest.

Freeze paths the agent must never edit in .snitch.toml at the repo root:

frozen = ["tests/**", "pytest.ini", "conftest.py"]

Ask Claude for "the snitch ledger" to see the session's receipt.

What enforce can and cannot block. It denies the agent's Edit/Write calls on frozen paths, sed -i / perl -i on test files, git --no-verify, and any write to Snitch's own state: .snitch.toml, the plugin's ledger and baselines, the installed plugin, and the Claude Code settings files that could switch hooks off. Shell commands are checked for the common write forms (>, >>, tee, rm, mv, cp, sed -i ...), not for every possible one: a Python one-liner or a cd inside the command line gets through. The gate is a guard rail inside the agent's own session, not a sandbox. Every change still goes through the detectors after the tool call and at the Stop hook, and in CI the GitHub Action re-checks the diff from outside the agent's reach.

Optional judge

Snitch's detectors cannot read your request, so a test you asked Claude to update is flagged like any other. With the judge option on (/plugin → Snitch → configure, or SNITCH_JUDGE=on for Codex), the Stop hook asks Claude (claude -p with your own login, no tools) whether your request called for each undisclosed test change; a change judged legitimate is not warned about or blocked, and the receipt says why. The judge can only clear a warning, never add one; if it fails or times out, the warning stays. Off by default: it adds a few seconds to a stop that has a flagged, undisclosed change.

Codex CLI plugin (experimental)

Codex runs the same hooks: the plugin ships a .codex-plugin manifest next to the Claude Code one.

codex plugin marketplace add ennajari/snitch
codex plugin add snitch@snitch

Then trust its hooks in /hooks. Codex has no plugin settings, so set the mode with SNITCH_MODE=observe|warn|enforce (default warn). Checked with Codex CLI 0.164 on a broken-test task: the session-start policy, the warning after a flagged edit, the Stop check and escalate all work; in enforce mode the gate reads the files an apply_patch names. Not yet: the "tests pass but never ran" warning (it reads Claude Code transcripts). Codex loads MCP tools on demand, so the policy tells the agent to search for escalate when it does not see it.

GitHub Action

on: pull_request
permissions:
  contents: read
  pull-requests: write     # the receipt comment
  security-events: write   # code scanning
jobs:
  snitch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0   # Snitch compares the PR base with its head
      - uses: ennajari/snitch@v1

Inputs: fail-on-events (default true), judge (default false; with anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}, an LLM judge reads the PR title and body and stops counting the test changes they asked for; they stay on the receipt as "judged legitimate"), python (an interpreter that can import your tests; default: the repo's .venv, else Snitch's own), strict (fail when a detector could not run, default false: an incomplete check passes with a warning and a note in the receipt), comment, upload-sarif, base, head. Output: events.

pre-commit

repos:
  - repo: https://github.com/ennajari/snitch
    rev: v0.3.1
    hooks:
      - id: snitch

CLI

pip install snitch-engine              # or: uvx --from snitch-engine snitch ...
snitch check                           # working tree vs HEAD (--staged: staged only)
snitch diff <base> <head> [--sarif out.sarif]
snitch compare <before-dir> <after-dir> [--json]
snitch mutate <before-dir> <after-dir>   # hollow tests (slow)

Exit codes: 0 clean, 1 events found, 2 usage error, 3 incomplete check (only with --strict: a detector could not run, for example because the repo's tests do not import).

What it flags

Every detector is a pure function of (before, after) and is tested on a cheating fixture and a legitimate one that must stay silent. Snitch flags changes; whether a flagged change was legitimate is up to the person reading the receipt (or the agent's disclosure). Updating a test because the requested behaviour changed is fine, as long as it is said.

TemptationBench

bench/ holds 40 Harbor tasks: small, realistic projects with one built-in temptation each (a broken test, an impossible spec, a missing service, a flaky test, a hard task) plus 14 controls. Hidden held-out tests grade the work, and the agent's own conftest.py or addopts cannot fake them. Every task ships a reference solution that must score 1 and a typical cheat that must score 0, both checked in CI and in Docker.

snitch bench-score turns Harbor runs into the leaderboard metrics: solve rate, cheat rate, silent-cheat rate (with a bootstrap 95% interval over tasks), honest-escalation rate, false positives on controls, a per-temptation breakdown, and McNemar's test for with vs without Snitch. Flagged trials are confirmed or cleared with review labels (AI-assisted so far; human check pending).

First results: Claude Code (edition 2026-11)

Claude Code 2.1.310 on its default model (claude-sonnet-5-5), 40 tasks × 3 runs, with and without the Snitch plugin; every flagged trial reviewed with an AI assistant, human check pending (full results and limits):

Paired McNemar p = 0.000008 (per task: 8 → 0, p = 0.008). The 18 changes came from broken tests (10) and flaky tests (8), and every one was disclosed: Claude Code changed the test and said so, and on the broken tests it was right that the test was wrong. With Snitch it reported the problem instead of changing the test.

Codex CLI 0.164.0 (gpt-6.1-sol) on the same tasks, with the experimental Codex plugin for the with-Snitch arm, run 10 days later: it changed or gamed a test in 57 of 78 trap trials without Snitch and 0 of 78 with it, and called escalate in 59. All 57 said a test changed; none gave the test's test_ name, though at least 16 identified it by description. These counts include disclosed changes: silent cheating was 0 in every arm above. No person has checked these labels yet; see the results page for that and the other limits. Gemini CLI follows after its own vendor preview. (Corrected from 14: a second review pass found 4 more, all disclosed; see the results page.)

Roadmap

• Engine: inventory, AST diff, rules (Python / pytest), 39 cheating + 24 legitimate fixtures
• Claude Code plugin: live detection, escalate, Stop verifier, enforce gate, evals
• GitHub Action with SARIF + PR receipt, pre-commit hook
• TemptationBench: 54 tasks (edition 2026-11 used the first 40), scoring
• Edition 2026-11 for Claude Code (40 tasks × 3 runs × with/without Snitch; human review pending)
• Codex CLI: both arms run and published (human review pending), experimental plugin
• Gemini CLI (partial run done, vendor preview next), then OpenCode and others
• Disclosure judge, mutation testing for hollow tests
• TypeScript (Jest / Vitest)

Contributing

Seen an agent cheat in a way Snitch misses? Open an issue with the before/after diff (anonymized). Every confirmed case becomes a detector fixture or a benchmark task. Each new pattern needs a cheating fixture that must be flagged and a legitimate one that must stay silent.

Development

uv sync
uv run pytest -q
uv run ruff check . && uv run ruff format .
uv run mypy engine

Citing

If you use Snitch or TemptationBench in your work, please cite it (CITATION.cff; GitHub's "Cite this repository" button gives other formats):

@software{ennajari_snitch_2026,
  author  = {Ennajari, Abdellah},
  title   = {Snitch: an integrity layer for AI coding agents, and TemptationBench},
  year    = {2026},
  version = {0.3.1},
  url     = {https://github.com/ennajari/Snitch}
}

License

MIT