🔥 hearsay

PR descriptions are hearsay. Verify them.

Coding agents write their own PR descriptions — and then reviewers merge based on them. "Updated all call sites." "Added tests for X." "No breaking changes." These are self-reported claims from the same model that wrote the code, and they are systematically overconfident. hearsay extracts every checkable claim from a PR description and verifies it against the actual diff with deterministic static analysis: regex, git plumbing, and the stdlib ast module.

No LLM. No API keys. No network. Zero runtime dependencies.

$ hearsay demo
hearsay — auditing 6 claim(s) in the PR description
base main (8b84bf5) → head HEAD (a261157) · 2 file(s) changed (+14/−1)

🔥 PANTS ON FIRE  “Renamed `greet` to `hello` and updated all call sites.”
   claimed complete, but 2 stale reference(s) to `greet` remain (`greet` → `hello`)
     - app.py:4  `print(greet("world"))`
     - test_greet.py:6  `greet.hello("world")`

❌ CONTRADICTED  “- Added tests for `hello`”
   1 test(s) exist but none of them can fail: test_greet.py: `test_greets_politely` — contains no assertions — it cannot fail

🔥 PANTS ON FIRE  “- No breaking changes”
   renamed public symbol(s) `greet` → `hello` without updating 2 call site(s)
     - app.py:4  `print(greet("world"))`

🔥 PANTS ON FIRE  “- Docs updated”
   the diff touches no documentation files (*.md, *.rst, docs/, CHANGELOG) — nothing was documented

⚠️ UNVERIFIABLE  “- Improved performance for hot paths”
   performance claims need benchmarks — no number, no claim

⚠️ UNVERIFIABLE  “Fixes #42”
   issue #42 cannot be fetched offline; the diff includes test changes, which is consistent with a fix

──────────────────────────────────────────────────────────────────────────────
6 claims audited: 1 contradicted · 3 pants-on-fire · 2 unverifiable

That whole audit ran offline against a synthetic repo in ~40 ms.

Why

• Agent-generated PR volume is exploding, and reviewers increasingly rubber-stamp the description instead of reading the diff.
• Research confirms the mismatch is systematic (see e.g. arXiv:2601.04886 on message–code inconsistency in agent commits).
• LLM-based reviewers can't be trusted to catch their own family's spin, cost money per call, and aren't reproducible. hearsay is the opposite: the same description + the same diff always produce the same verdicts.

Install

pipx install hearsay-pr     # or: uv tool install hearsay-pr
# the PyPI distribution is `hearsay-pr`; the CLI command it installs is `hearsay`
# or, from a clone of this repo:
pip install .
# or, for development:
PYTHONPATH=src python -m hearsay demo

Requires Python 3.10+ and git. No other dependencies.

Why deterministic, not LLM?

LLM reviewers are useful; they are just not evidence. hearsay is the deterministic layer underneath them.

How it works

PR description (text)
  │  31 builtin regex patterns ─ active/passive/headline voice
  ▼
structured claims ──► one deterministic checker per claim type
  │                     │ git plumbing (cat-file --batch)
  │                     │ stdlib ast (Python) / symbol tables (JS/TS, Go)
  │                     │ repo-wide stale-reference & API-surface scans
  ▼
verdicts: ✅ corroborated · ❌ contradicted · 🔥 pants-on-fire · ⚠️ unverifiable
  ▼
terminal report · PR comment (markdown) · JSON for your bot

Usage

# audit a local branch against main, description from a file
hearsay check --base main --desc pr.md

# ...or read the description from stdin (e.g. your editor's PR draft)
hearsay check --base main --desc - < description.md

# ...or pull it straight from GitHub via the gh CLI
hearsay check --pr 123 --base origin/main

# CI gate: exit 2 on pants-on-fire, exit 1 on contradicted
hearsay gate --pr 123 --base origin/main --fail-on contradicted

# post the report as a PR comment (uses gh for auth; hearsay never sees tokens)
hearsay gate --pr 123 --base origin/main --comment

# machine-readable output for your bot
hearsay check --pr 123 --format json
hearsay check --pr 123 --format markdown   # paste into a PR comment

# project setup & introspection
hearsay init        # write a commented hearsay.toml
hearsay patterns    # list the builtin claim patterns

Analysis engines (beyond the PR gate)

The claim gate checks one PR. These three engines audit the whole repository — they're what makes hearsay a quality engine rather than a linter for prose:

hearsay testlint              # six test-smell families, per-test findings
hearsay testlint --fail-on vacuous,no-production   # gate on any mix
hearsay score                 # deterministic test-health score, 0-100, A-D
hearsay score --min 70        # ratchet gate: fail below the bar
hearsay clones                # copy-paste clone clusters (the GitClear #1 metric)

testlint finds: tests that cannot fail, assertions that pin almost nothing (is not None, truthiness-only), byte-identical duplicate test bodies (agent filler), tests that never touch production code, sleep-based flaky tests, and public functions no test references anywhere. score compiles those into one number with a full breakdown — every deduction is a counted issue family, never vibes. All engines are offline, deterministic, and JSON-serializable.

hearsay mutiny — does your test suite actually notice?

Static analysis approximates; mutation testing proves. hearsay mutiny applies one deterministic AST mutation at a time to production code (+→-, ==→!=, True→False, dropped return values…), runs your tests, and lists every change your suite cannot detect:

hearsay mutiny --max-mutants 30 --fail-on-survived

Python-only, opt-in (it really runs your tests), and the strongest answer to "is this test suite real?".

Type-3 clones

hearsay clones --normalize-idents additionally matches clones whose identifiers were renamed — the agent copy-rename-duplicate signature.

Extend it: plugins

# hearsay.toml
plugins = ["hearsay_plugin.py"]

A plugin module provides extra PATTERNS (claim regexes) and CHECKERS (verdict functions) for existing claim kinds — same contract, same design rules. See CONTRIBUTING.md.

Forge-agnostic PR retrieval

--pr-cmd "glab mr view {pr}" (or any command with a {pr} placeholder whose stdout is the description) replaces the default gh integration — GitLab, Gitea, or your internal tooling.

The Hearsay Report generator

hearsay report --cases cases.jsonl --out report.md turns labeled calibration cases into a shareable markdown report — headline precision, per-claim-kind results, confirmed fire samples. That's the quarterly publication pipeline, as a command.

Let the agent police itself (MCP)

hearsay mcp serves the engines as MCP tools over stdio — no dependencies, hand-rolled JSON-RPC. Register it in your MCP client (or just add one line to AGENTS.md):

{"hearsay": {"command": "hearsay", "args": ["mcp"]}}

<!-- AGENTS.md -->
Before opening a PR, run hearsay_check on your own description and fix
every pants-on-fire claim. Do not claim work you cannot demonstrate.

The agent that wrote the slop becomes the first line of defense against it.

Fact-check commit messages too

Agents overclaim in commit messages exactly like PR descriptions — and commit messages can be gated locally, no PR required:

hearsay check --base main --commits        # audit the branch's messages

# .git/hooks/commit-msg — block provable lies before they land
hearsay commitmsg "$1"

commitmsg compares the message against the staged diff and defaults to --fail-on pants-on-fire: hooks only block what can be proven.

The ratchet: adopting score on imperfect repos

Absolute thresholds are why quality gates get uninstalled. hearsay score supports a baseline ratchet instead:

hearsay score --save-baseline    # writes .hearsay-baseline.json — commit it
hearsay score                    # passes at the baseline, fails on regression

Failures show per-issue-kind deltas ("vacuous: 2 → 5"), so the gate tells you exactly what got worse. Fix forward, re-baseline when you've earned it.

Configuration

hearsay init writes a commented hearsay.toml; the same keys work under [tool.hearsay] in pyproject.toml (hearsay.toml wins):

# hearsay.toml
base = "origin/main"       # CLI --base overrides this
fail_on = "contradicted"   # pants-on-fire | contradicted | unverifiable | none
format = "terminal"        # terminal | markdown | json
exclude = ["**/generated/**", "vendor/**"]   # ignored everywhere

Precedence: CLI flag > hearsay.toml > pyproject [tool.hearsay] > builtin default.

What gets checked

Verdicts

The three-valued logic is the point: a checker that never says "unverifiable" will eventually say "corroborated" about a lie.

Language support

hearsay is precise where it can be and honest where it can't:

Adding a language is a self-contained Language(...) entry plus tests — a deliberate good-first-issue (see CONTRIBUTING.md).

Use it as a GitHub Action

name: hearsay
on: [pull_request]
jobs:
  factcheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }   # hearsay needs the merge base
      - uses: hearsay-dev/hearsay@v0
        with:
          base: origin/main
          fail-on: contradicted    # pants-on-fire always fails
          comment: true            # post the report to the PR

Or roll your own step — hearsay is a plain CLI:

      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hearsay-pr
      - env: { GH_TOKEN: "${{ github.token }}" }
        run: hearsay gate --pr "${{ github.event.pull_request.number }}" --base origin/main

Exit codes: 0 pass · 1 contradicted/unverifiable (per --fail-on) · 2 pants-on-fire · 3 usage or git error.

Use with pre-commit

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/hearsay-dev/hearsay
    rev: v0.3.0
    hooks:
      - id: hearsay-gate
        args: ["--base", "origin/main", "--desc", "pr-description.md"]

pre-commit run hearsay-gate --hook-stage manual

Design principles

1. Deterministic. Same inputs, same verdicts, always. Auditable regex and AST checks — you can read every rule in src/hearsay/patterns.py and src/hearsay/checkers.py.
2. Offline and free. Zero runtime dependencies, zero API calls, runs anywhere git runs.
3. Three-valued logic. "We don't know" is a first-class verdict.
4. Conservative by default. Claims that can't be proven false are never marked false. --fail-on lets you choose how paranoid the gate is.
5. Evidence or it didn't happen. Every non-corroborated verdict cites file:line locations you can click.

Calibration (why there's no precision badge yet)

Unit tests prove the checkers work on fixtures. They don't prove the false-positive rate — only labeled real-world PRs do. hearsay ships the whole calibration pipeline:

hearsay calibrate --cases cases.jsonl                        # batch predict
hearsay calibrate --cases cases.jsonl --emit-worksheet ws.md # label by hand
hearsay calibrate --cases cases.jsonl --json > results.json  # score

Protocol and acceptance thresholds: docs/CALIBRATION.md. Until fire precision ≥ 97% on ≥ 100 labeled claims, the README will not recommend gating on contradicted — and the badge stays honest.

Roadmap

• Multi-language tiering (Python precise; JS/TS, Go heuristic)
• Calibration pipeline (hearsay calibrate + protocol)
• Reach fire precision ≥ 97% on ≥ 100 labeled real PRs (needs you — see docs/CALIBRATION.md)
• Signature-change detection (breaking positional-arg changes)
• tree-sitter backends to promote more languages to the precise tier
• hearsay report — aggregate scan of N repos → the periodic Hearsay Report on how often agent PR descriptions survive fact-checking
• PR-comment bot mode with persistent verdict tables
• Plugin API for community checkers

FAQ

Doesn't the regex extraction miss things? Yes — that's why unverifiable exists. hearsay only rules against a claim when a deterministic check proves it, so misses are safe. Template-friendly descriptions (headings, bullets) extract best.

Is this accusing agents of lying? No — agents are overconfident, not malicious. hearsay treats the description as untrusted input, exactly like you treat user input.

Why not just use an LLM reviewer? See Why deterministic, not LLM? — reproducibility, cost, and independence. hearsay has no incentive to believe its own kind.

Development

pip install -e .[dev]     # or: pip install ruff mypy coverage
ruff check src tests
mypy
python -m unittest discover -s tests -t .   # PYTHONPATH=src if not installed
python scripts/bench.py   # performance smoke test on a synthetic 1200-file repo

The full test suite is green, ruff + mypy clean. New contributors: CONTRIBUTING.md doubles as the architecture tour, and good first issue = add a language adapter.

License

MIT — see LICENSE.