bite-check 0.4.1

Bite checks whether an AI coding agent's "fixed" is actually true.

"Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that should exist for every "fixed" commit."

— Ivan Zdravkov, on DEV

When a commit says it fixed something, Bite writes one small test of that exact claim, runs it itself on the commit and on today's code, and shows the result only if a second, independent check agrees the test is fair. If the test fails, the "fixed" wasn't.

Try it in a minute

Install once with pipx, which gives Bite its own space so there's no venv to set up or activate:

python -m pip install --user pipx
python -m pipx ensurepath

Close the terminal window and open a new one, then:

pipx install bite-check
cd /path/to/your-repo
bite --free

bite now works in any terminal window. bite --free needs no key and costs nothing. It takes each recent change back out, runs your tests, and lists the changes no test guards, plus any new test that still passes without its commit's code (it tests nothing). If something it needs is missing, like git or pytest, it says so and gives you the line that installs it. Each run ends with a one-page report, bite_proof/report.html, to open in your browser. To update Bite later: pipx upgrade bite-check.

To test the claims themselves (the paid step), add one extra package and set ANTHROPIC_API_KEY, then run bite:

pipx inject bite-check claude-agent-sdk

This step uses your own API key, and --budget caps what a run spends ($1 by default). It runs on Anthropic's Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule: https://code.claude.com/docs/en/agent-sdk/overview). The commits Bite checks can come from any coding AI or person.

Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: AGENTS.md, written so your coding AI (Claude Code, Codex, Cursor, Copilot) can set Bite up for you. To work on Bite itself: clone it, pip install -e .[test], then python -m pytest.

Run it on every pull request

Add this file to your repo as .github/workflows/bite.yml. On each pull request it runs the free check on that pull request's commits and shows a pass or fail check next to your other CI. It fails when a commit's change can be taken back out and your tests still pass. No key, no cost. It's also listed on the GitHub Marketplace as Bite free check.

name: bite
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  bite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      # install what your tests need here, e.g. for Python:
      - uses: actions/setup-python@v5
      - run: python -m pip install pytest -r requirements.txt
      - uses: connabyss/bite@v0.4.0

Options go under with:: args (extra bite options, e.g. --serious), max-commits (default 20), fail-on-gap (set to false to report without failing) and comment (set to false for no comment). It posts one comment on the pull request that says, per commit, whether a test noticed the change being taken out, and updates that same comment on later pushes. The rerun steps for each flagged commit are on the job's summary page.

Bite-proof badge

When every change Bite checked had a test that noticed it, Bite ends with "Bite-proof. Nice work." and prints a badge line you can paste into your README:

Prefer a gentler wording for the warnings? bite --roast gives the same results, phrased differently.

The catch that shows what it's for

In fellowapp/prosemirror-py, commit 3614c96 (co-authored by Claude) says:

fix: add weakref callback to clean up resolve cache entries

The cache used dict keyed by id(doc) with no cleanup, causing a slow memory leak in long-running processes. Added a weakref finalizer callback that removes the cache entry when the doc is garbage collected.

Bite wrote this test of that claim and ran it:

def test_cache_entry_removed_after_doc_gc():
    resolvedpos._resolve_cache.clear()
    d = doc(p("hello"))
    ResolvedPos.resolve_cached(d, 2)
    assert len(resolvedpos._resolve_cache) == 1
    del d
    gc.collect()
    assert len(resolvedpos._resolve_cache) == 0   # fails: 1 == 0

It fails on that commit and on today's code. The cache keeps each document alive through its cached positions, so the cleanup never runs. After 20,000 documents are resolved and dropped, the cache still holds all 20,000, using 35 MB. The commit looked right, read right, and passed its own tests. Reported with a repro in #335; a tested fix is open as #336.

Fixed or merged upstream

Each bug was reproduced with a standalone script in a clean environment before it was reported.

Sent, waiting for review

Limits, plainly

• Most claims hold. Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.
• Quiet is not a pass. Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.
• It needs the project's tests to run. If they don't run on a clean copy, Bite can't check that commit.
• Bug hunting is weaker than claim checking. In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.
• Small samples, AI judges. The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.

Contributing

Bug reports, fixes and AI-assisted PRs are all welcome. See CONTRIBUTING.md. If Bite flagged something wrongly, or missed a claim you know was false, please open an issue.

Thank you to Bite's first contributors:

• @snehaapratap made the setup error messages point to steps that exist (#18).
• @gauravjangra3165 added macOS to the tests (#19).
• @bfs2021 made the "check it yourself" steps work for commits that add a new file (#37, fixing #27).

MIT licensed.