bite-check 0.7.0

Bite checks whether an AI coding agent's "fixed" is actually true.

It takes each recent fix back out of your code and runs your tests. If they still pass, nothing guards that fix, and Bite tells you which one. That free check needs no key and works with pytest, npm test, cargo test, go test and dotnet test:

pipx install bite-check
bite --free

With your own API key it goes one step further: it writes one small test of the commit's exact claim and runs it. That is how it found a memory leak in prosemirror-py that a commit said it had fixed.

"Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that should exist for every "fixed" commit."

— Ivan Zdravkov, on DEV

Try it in a minute

Install once with pipx, which gives Bite its own space so there's no venv to set up or activate:

python -m pip install --user pipx
python -m pipx ensurepath

Close the terminal window and open a new one, then:

pipx install bite-check
bite --demo

bite --demo builds a tiny made-up shop project in a temporary folder and runs the free check on it. Two of its commits say they fixed something: one added a test for its fix, the other didn't. Bite takes each fix back out and runs the tests, and you see the difference:

  9661edcf ✅ noticed by test_discount_never_below_zero Fix discounts over 100% making the price negative
  e5c329ec ⚠️ NO TEST NOTICED           Fix totals showing long decimals like 0.30000000000000004

It needs nothing besides git, takes a few seconds, and doesn't touch your files. Then try it on your own project:

cd /path/to/your-repo
bite --free

bite now works in any terminal window. bite --free needs no key and costs nothing. It takes each recent change back out, runs your tests, and lists the changes no test guards, plus any new test that still passes without its commit's code (it tests nothing). If something it needs is missing, like git or pytest, it says so and gives you the line that installs it. Each run ends with a one-page report, bite_proof/report.html, to open in your browser. To update Bite later: pipx upgrade bite-check.

To test the claims themselves (the paid step), add one extra package and set ANTHROPIC_API_KEY, then run bite:

pipx inject bite-check claude-agent-sdk

This step uses your own API key, and --budget caps what a run spends ($1 by default). It runs on Anthropic's Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule: https://code.claude.com/docs/en/agent-sdk/overview). The commits Bite checks can come from any coding AI or person.

Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: AGENTS.md, written so your coding AI (Claude Code, Codex, Cursor, Copilot) can set Bite up for you. To work on Bite itself: clone it, pip install -e .[test], then python -m pytest.

Check Claude's "fixed" live, in Claude Code

Bite can also check the moment Claude says it fixed something, before you read "fixed!". Each time Claude finishes a turn that changed code, Bite takes the change back out of a copy of your project and runs your tests. If none of them notice, Claude is sent back once to add a test that fails without its change. If it stops again and still no test notices, you get a warning instead, so it never goes round in circles. It's the free check: no API key.

In Claude Code, type:

/plugin marketplace add connabyss/bite
/plugin install bite@bite

and install Bite itself if you haven't (bite live is new: until the next release, install it from GitHub):

pipx install git+https://github.com/connabyss/bite

/bite checks the changes since your last commit whenever you like. In a trial, Claude fixed a discount and then wrote assert total([50, 50], 10) == 90. That passes with or without the fix (100 - 10 and 100 × 90% are both 90), and Bite said so:

Bite ⚠️ the tests that came with this change (test_cart.py) don't notice it: with the change to cart.py taken out,
they still pass. The "fixed" isn't proven yet.

Without the plugin, run bite live yourself after a change, or bite live --hook from any tool that runs a command when the AI stops (it reads Claude Code's Stop hook JSON).

Run it on every pull request

Easiest: install the bite-check app on the repos you want. Within an hour it opens one issue there with an "Add Bite" link that fills in the file below for you; you read it and click Commit changes. The app can only open issues: it can't read or change your code or workflows.

Or add this file to your repo yourself as .github/workflows/bite.yml. On each pull request it runs the free check on that pull request's commits and shows a pass or fail check next to your other CI. It fails when a commit's change can be taken back out and your tests still pass. No key, no cost. It's also listed on the GitHub Marketplace as Bite free check.

name: bite
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  bite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      # install what your tests need here, e.g. for Python:
      - uses: actions/setup-python@v5
      - run: python -m pip install pytest -r requirements.txt
      - uses: connabyss/bite@v0.7.0

Options go under with:: args (extra bite options, e.g. --serious), max-commits (default 20), fail-on-gap (set to false to report without failing) and comment (set to false for no comment). It posts one comment on the pull request that says, per commit, whether a test noticed the change being taken out, and updates that same comment on later pushes. The rerun steps for each flagged commit are on the job's summary page.

Have Bite write the missing test (opt-in, your own key)

When no test noticed a change, Bite can write that test for you. Add your Anthropic API key as a repository secret named ANTHROPIC_API_KEY, then:

      - uses: connabyss/bite@v0.7.0
        with:
          write-tests: true
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          args: --budget 0.50

For each change no test noticed (3 at most, --suspects changes that), a fresh AI worker writes one small test on a copy of the project. Bite then runs it itself: it must pass twice with the change in, and fail with the change taken out. Only then does it show up in the comment, folded, ready to paste in. Otherwise Bite throws it away and says so. --budget caps what a run spends ($1 by default). Pull requests from forks don't get secrets, so on those Bite runs the free check only. On your own computer: bite --write-tests.

Every test Bite keeps is also a download: bite-missing-tests under Artifacts on the run's page, each file already in the folder it belongs in.

One click instead of a paste (opt-in): add open-pr: true and Bite opens a small pull request into your pull request's branch with the test in it. Click Merge there and the test is in. It runs on your workflow's own token, not on any app, so your workflow needs:

permissions:
  contents: write        # Bite puts the test on a branch of its own, bite/missing-tests-<number>
  pull-requests: write

and, in your repository, Settings, Actions, General: tick Allow GitHub Actions to create and approve pull requests. Without that tick, Bite still puts the test on its branch and the comment links to open the pull request yourself. GitHub doesn't run your checks on a pull request a workflow opened, so they run once you merge it into your branch; Bite has already run the test both ways. Not on pull requests from forks.

Bite-proof badge

When every change Bite checked had a test that noticed it, Bite ends with "Bite-proof. Nice work." and prints a badge line you can paste into your README:

Prefer a gentler wording for the warnings? bite --roast gives the same results, phrased differently.

Famous repos Bite has checked

Checked Oct 30-31, 2026 with bite --free. Every bug fix Bite could check had a test (one of chi's only works in a terminal). We read each unnoticed change by hand. Apart from the gaps above, they were things a test can't catch, like type hints, speed-ups, Windows-only code or rewrites that work the same.

The full scoreboard has 20 projects in 5 languages, with a link to every gap.

The catch that shows what it's for

In fellowapp/prosemirror-py, commit 3614c96 (co-authored by Claude) says:

fix: add weakref callback to clean up resolve cache entries

The cache used dict keyed by id(doc) with no cleanup, causing a slow memory leak in long-running processes. Added a weakref finalizer callback that removes the cache entry when the doc is garbage collected.

Bite wrote this test of that claim and ran it:

def test_cache_entry_removed_after_doc_gc():
    resolvedpos._resolve_cache.clear()
    d = doc(p("hello"))
    ResolvedPos.resolve_cached(d, 2)
    assert len(resolvedpos._resolve_cache) == 1
    del d
    gc.collect()
    assert len(resolvedpos._resolve_cache) == 0   # fails: 1 == 0

It fails on that commit and on today's code. The cache keeps each document alive through its cached positions, so the cleanup never runs. After 20,000 documents are resolved and dropped, the cache still holds all 20,000, using 35 MB. The commit looked right, read right, and passed its own tests. Reported with a repro in #335; a tested fix is open as #336.

Fixed or merged upstream

Each bug was reproduced with a standalone script in a clean environment before it was reported.

Sent, waiting for review

Limits, plainly

• Most claims hold. Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.
• Quiet is not a pass. Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.
• It needs the project's tests to run. If they don't run on a clean copy, Bite can't check that commit.
• Bug hunting is weaker than claim checking. In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.
• Small samples, AI judges. The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.

Contributing

Bug reports, fixes and AI-assisted PRs are all welcome. See CONTRIBUTING.md. If Bite flagged something wrongly, or missed a claim you know was false, please open an issue.

Thank you to Bite's first contributors:

• @snehaapratap made the setup error messages point to steps that exist (#18).
• @gauravjangra3165 added macOS to the tests (#19).
• @bfs2021 made the "check it yourself" steps work for commits that add a new file (#37, fixing #27).

MIT licensed.