RunBoth

An AI changed your code. RunBoth runs both versions and tells you what actually behaves differently, including the functions nobody touched.

runboth.dev · How it works · Red team results

$ git commit -m "refactor: tidy up the rates module"

  BLOCKED: the behaviour changed and your message does not say so.

    rate(100)
      used to:  return 0.1
      now:      return 0.0

  AND 1 function you did NOT touch now behaves differently,
  because it calls what you changed:

    total(2.5, 100)          pkg/invoice.py
      used to:  return 225.0
      now:      return 250.0

Why this exists

An AI refactors a module and reports that the behaviour is unchanged. Usually that is true. Measurably often it is not: Dristi and Dwyer tested six models across three datasets and two refactoring types and found LLM refactorings functionally non-equivalent 19 to 35% of the time (arXiv:2602.15761, February 2026).

Review does not catch it. A diff is an honest record of the edit, not of the effect, and those are different documents. The function that broke is often in a file the diff never mentions, because it only calls what changed.

Tests do not close the gap either. A test asserts what somebody previously thought to assert, and the code most likely to move silently is the code nobody wrote a test for. If an AI wrote the tests as well, you have asked one system to mark its own homework.

So RunBoth does the boring thing that works: it runs both versions and compares what actually comes out.

What it does

It checks out both versions of your code, generates inputs for every changed function from its signature and from the constants mined out of its own bytecode, runs both versions in separate sandboxed subprocesses, and compares seven observation channels. When they disagree it hands you the exact input that separates them.

No test suite required. No network calls. No AI model. No dependencies.

Install

pip install runboth                                  # once the first release is on PyPI
pip install git+https://github.com/runboth/runboth   # works today
runboth install-hook          # a commit-msg gate, silent unless behaviour moved

As a GitHub Action, running on your own runners:

- uses: runboth/runboth@v0.1.0
  with:
    budget: 60

Three verdicts, never two

"Cannot tell" and "no difference" are different claims, and collapsing them into a green check is how tools end up lying. RunBoth never says safe.

The seven channels

Return value · exception raised · warnings · stdout · stderr · argument mutation · object state.

A narrow definition of behaviour does not under-report, it lies, because whatever sits outside the definition comes back as no_change.

What it is not

Not a model checker. Kani and CBMC translate code into logic, let inputs be unconstrained symbols, and ask a solver whether a bad state is reachable within a bound. They return a proof. RunBoth executes real code on concrete values. It finds differences and reproduces them; it cannot prove absence, and never claims to.

Not mutation testing. Mutation testing damages your code to score your test suite. RunBoth damages nothing; both versions come from your git history, and no test suite is needed.

Measured

Red-teamed against eight public repositories it had never been tuned on, with an automated oracle built to catch the tool lying. 2,548 functions, zero false positives. Full method and numbers in RED_TEAM_2026-09-12.md.

An adversarial corpus of 22 functions written specifically to induce false positives (object addresses in default repr, datetime.now, unseeded random, uuid4, os.getpid, set iteration order, mutable defaults, generators, __file__ paths) produced none.

Environment variables

git commit --no-verify also bypasses the gate, and the gate says so itself when it blocks.

Honest limits

• Function-level checking is Python only. Changed files in other languages are named explicitly rather than passed over quietly.
• Sampling finds differences; it cannot prove their absence.
• Nondeterministic, too-slow, or unconstructible functions abstain with a reason, and are never counted as passing.
• The sandbox contains accidents: resource limits, network blocked, filesystem writes blocked. It is not a security boundary against hostile code, and no pure-Python sandbox is.
• A _version.py that the build generates is synthesised as 0.0.0 so the package can be imported at all. A commit that changes how a version string is derived is therefore not measured. Both sides get the same stub, so it can never manufacture a changed, and RUNBOTH_NO_VERSION_STUB=1 turns it off.

Development

pip install -e .
pytest tests/ -q
runboth selftest          # the control suites, half of which must fail

Licence

FSL-1.1-Apache-2.0. Free for every use except building a competing product, and it converts to plain Apache 2.0 two years after each release.

Built by Kyle Clouthier at Clouthier Simulation Labs.