RunBoth

An AI changed your code. RunBoth runs both versions and tells you what actually behaves differently, including the functions nobody touched.

runboth.dev · How it works · Red team results

$ git commit -m "refactor: tidy up the rates module"

  BLOCKED: the behaviour changed and your message does not say so.

    rate(100)
      used to:  return 0.1
      now:      return 0.0

  AND 1 function you did NOT touch now behaves differently,
  because it calls what you changed:

    total(2.5, 100)          pkg/invoice.py
      used to:  return 225.0
      now:      return 250.0

What it does

It checks out both versions of your code, generates inputs for every changed function from its signature and from the constants mined out of its own bytecode, runs both versions in separate sandboxed subprocesses, and compares seven observation channels. When they disagree it hands you the exact input that separates them.

No test suite required. No network calls. No AI model. No dependencies.

Install

pip install runboth                                  # once the first release is on PyPI
pip install git+https://github.com/runboth/runboth   # works today
runboth install-hook          # a commit-msg gate, silent unless behaviour moved

As a GitHub Action, running on your own runners:

- uses: runboth/runboth@v0.1.0
  with:
    budget: 60

Three verdicts, never two

"Cannot tell" and "no difference" are different claims, and collapsing them into a green check is how tools end up lying. RunBoth never says safe.

The seven channels

Return value · exception raised · warnings · stdout · stderr · argument mutation · object state.

A narrow definition of behaviour does not under-report, it lies, because whatever sits outside the definition comes back as no_change.

What it is not

Not a model checker. Kani and CBMC translate code into logic, let inputs be unconstrained symbols, and ask a solver whether a bad state is reachable within a bound. They return a proof. RunBoth executes real code on concrete values. It finds differences and reproduces them; it cannot prove absence, and never claims to.

Not mutation testing. Mutation testing damages your code to score your test suite. RunBoth damages nothing; both versions come from your git history, and no test suite is needed.

Measured

Red-teamed against eight public repositories it had never been tuned on, with an automated oracle built to catch the tool lying. 2,548 functions, zero false positives. Full method and numbers in RED_TEAM_2026-09-12.md.

An adversarial corpus of 22 functions written specifically to induce false positives (object addresses in default repr, datetime.now, unseeded random, uuid4, os.getpid, set iteration order, mutable defaults, generators, __file__ paths) produced none.

Environment variables

git commit --no-verify also bypasses the gate, and the gate says so itself when it blocks.

Honest limits

• Function-level checking is Python only. Changed files in other languages are named explicitly rather than passed over quietly.
• Sampling finds differences; it cannot prove their absence.
• Nondeterministic, too-slow, or unconstructible functions abstain with a reason, and are never counted as passing.
• The sandbox contains accidents: resource limits, network blocked, filesystem writes blocked. It is not a security boundary against hostile code, and no pure-Python sandbox is.

Development

pip install -e .
pytest tests/ -q
runboth selftest          # the control suites, half of which must fail

Licence

FSL-1.1-Apache-2.0. Free for every use except building a competing product, and it converts to plain Apache 2.0 two years after each release.

Built by Kyle Clouthier at Clouthier Simulation Labs.