moonbuggy

Fast, agent-first mutation testing for Python.

Mutation testing measures whether your tests would actually notice if the code broke. It makes small changes to your source — flipping a < to a <=, a True to a False — and reruns the tests. A change no test objects to is a gap: a missing test, a weak assertion, or a line nothing exercises.

Two things make moonbuggy different:

• Speed. It runs only the tests that actually cover each mutated line, applies mutations in memory rather than writing files, runs mutants in parallel forked processes, does the work every mutant shares once in a warm host rather than once per mutant, and caches results across runs. On a suite where test execution dominates, that is 38x faster than the naive approach of rerunning everything per mutant, and about 1.9x faster than mutmut. Both figures come from make bench, which measures all three tools in one session on one machine; see benchmark results for what they do and do not establish.
• Output built for agents. Results are JSON Lines, with a derived plaintext view whose every line starts with a fixed keyword, so grep SURVIVED works with no knowledge of the schema.

Install

Full documentation is published at jhamon.github.io/moonbuggy.

Requires Python 3.12+ and pytest.

pip install .

Use

From your project root, with no flags and no configuration:

moonbuggy

It discovers your source layout, runs one instrumented pass to build a line→test map, then runs each mutant against only its covering tests.

Two files land in .moonbuggy/:

Exit code is 0 when nothing survived, 1 when there are survivors, and 2 when the run could not start.

Reading the output

Every plaintext line begins with one of exactly five keywords:

So the thing you usually want is:

grep SURVIVED .moonbuggy/results.txt

Each line carries key=value tokens, including nearest_test= — the test to extend to close that gap.

Lines are one per mutant and never contain the diff, so they stay grep- and awk-friendly. To see a mutant in full:

moonbuggy show 'shipping.py:5:comparison_swap:0'

Suppressing an equivalent mutant

Some mutants cannot be killed by any test because the mutated program is genuinely equivalent — changing a cache size, say. Detecting these automatically is undecidable, so moonbuggy does not try. Mark them in the source:

CACHE_SIZE = 128  # moonbuggy: skip -- tuning only, no observable behaviour

They are then reported SKIPPED rather than silently dropped, so the mutant count stays honest.

Options

Nothing below is required.

--timeout SECONDS    before a mutant is called TIMEOUT (default: 30)
--operators NAMES    comma-separated subset, e.g. comparison_swap,boundary
--include FRAGMENT   only mutate paths containing FRAGMENT (repeatable)
--exclude FRAGMENT   skip paths containing FRAGMENT (repeatable)
--jobs N             mutants to run concurrently (default: CPU count - 1)
-n, --workers N      pytest-xdist workers per mutant run
--source DIR         directory to mutate, if discovery guesses wrong
--no-cache           ignore and do not update the cache
--clear-cache        delete the cache, then run
--quiet              summary line only

Development

python -m venv .venv && .venv/bin/pip install -e '.[dev,bench]'

The project under tests/fixtures/sample_project is input data, not tests of moonbuggy — a small pytest project whose 22 mutants have hand-written expected outcomes in oracle.toml. Some of its tests hang or fail by design once mutated, which is why the outer suite excludes it.

Status

Phase 0 and Phase 1 of the acceptance criteria are implemented and all criteria are met. Speed numbers and the four measured iterations behind them are in docs/benchmark-results.md.

Design notes: spike A (in-memory mutation, xdist), spike B (coverage mechanism).