Cerebrum

LLM-based mutation testing, portable to any codebase.

Cerebrum inserts small, realistic bugs ("mutations") into a codebase one at a time, runs the project's test suite, and reports whether the tests caught each mutation. Mutations that survive reveal gaps in the test suite — each one is a concrete bug that would ship silently.

The name is a nod to Cerebro, the machine in X-Men used to find mutants.

Why another mutation tester?

Traditional mutation testers (Stryker, PIT, mutmut, cosmic-ray) rely on a per-language AST library with a fixed set of operators (+→-, <→<=, …). Cerebrum uses an LLM as the mutation operator, which:

• works on any language without a per-language mutator, and
• produces domain-realistic bugs that resemble real developer mistakes, stressing the test suite more meaningfully than mechanical operators.

Core design: Engine + Adapter

• The Engine is language-agnostic. It knows git, patches, running a command, and comparing exit codes — never npm, jest, or pytest.
• Everything codebase-specific lives in one cerebrum.yaml committed to the target repo. Porting Cerebrum to a new codebase = writing that one file.

        cerebrum.yaml (per target repo)
        modules · install · test · globs
                     │ read by
        ┌────────────▼─────────────────────────────────────────┐
        │                 CEREBRUM ENGINE                       │
        │  1. BASELINE   install → run suite → require green    │
        │  2. TARGETING  pick lines (coverage | changed | ...)  │
        │  3. GENERATE   LLM → ONE mutant patch + metadata      │
        │  4. EXECUTE    worktree: apply → build → test → class │
        │  5. REPORT     score · survivors · suggested tests    │
        │                · severity-weighted trend across runs  │
        └───────────────────────────────────────────────────────┘

Key principles

• One mutant at a time in an isolated git worktree — enables correct attribution of a test failure to a specific mutation.
• Pure-LLM operator behind a validity gate: a mutant only counts if its patch applies, changes behavior, and builds/lints.
• Coverage-guided targeting — never mutate uncovered lines (they survive trivially and add noise). When a run's mutant cap can't cover every eligible line, selection is distributed fairly across files (no single file can exhaust the cap while others get none) and randomized per run, so trending scores over many runs converge on a representative picture of the whole module instead of a fixed, alphabetically-biased subset.
• One targeting vocabulary: cerebrum run sweeps a module using the strategy named in config (coverage by default); `cerebrum run --diff ..` mutates only lines changed in that range (PR gate). The CLI never invents its own strategy names — `--diff` only supplies the range a committed config can't.

Post-run hook

cerebrum.yaml accepts an optional top-level after_run: <command> — a single shell command the engine runs, in the repo root, once a run's .cerebrum/ artifacts (history.sqlite, runs/<run_id>/mutants.jsonl, runs/<run_id>/coverage.json) have been written. Use it to push results somewhere (e.g. after_run: cerebrum-cloud-push after pip install cerebrum-cloud-push — see integrations/cerebrum-cloud/ for the Cerebrum Cloud client) without teaching the engine anything about the destination. It's best-effort: a failing after_run command logs a warning to stderr but never fails the run or changes its exit code or reported mutation score.

Setup

Install the engine: pip install cerebrum-engine. If you're working in this repo instead, install from the repo root with the dev/test toolchain included: pip install -e ".[dev]".

Cerebrum needs ANTHROPIC_API_KEY set in the process environment for mutation generation and severity scoring — it only ever reads os.environ["ANTHROPIC_API_KEY"] and has no opinion on how that variable gets there.

Avoid putting the raw key in a plaintext .env — copy .env.example to .env and put a secrets-manager reference in it instead, e.g. for 1Password:

ANTHROPIC_API_KEY=op://<vault>/<item>/credential

Then run through that tool so it resolves the reference into the cerebrum child process only, never onto disk:

op run --env-file=.env -- cerebrum run -c cerebrum.yaml --module backend

The same pattern works with Doppler, Vault, or any other secrets manager — Cerebrum doesn't care which one, as long as the real value lands in the process environment before it runs. Don't bake a specific secrets tool into shared scripts or docs; each user or environment should be free to supply the key however they manage secrets.

Status

Early, but the core loop runs. The config adapter, baseline stage, and the single-mutant lifecycle (cerebrum mutate) are implemented: it selects a covered line, asks Claude to insert one bug, applies it in an isolated git worktree, runs the suite, classifies the outcome, and appends a record to .cerebrum/mutants.jsonl. Generating a real mutant needs ANTHROPIC_API_KEY (see Setup). Targeting and sweeps (cerebrum run, --diff) are implemented: pick covered lines via the config's strategy or a changed-lines diff range, then mutate them in parallel across a pool of pre-installed, reused git worktrees (runtime.parallelism) instead of one at a time. Reporting (cerebrum report) is implemented: mutation score, a survivor report with LLM-suggested tests, and a severity-weighted trend across runs. The llm-risk targeting strategy is implemented: within each file's already-fair share of the budget, it weights which lines get picked toward ones Claude judges higher-risk (complex conditionals, edge cases, financial/security logic), at the cost of up to one extra LLM call per scored file per run — a file falls back to uniform weighting automatically if its call fails. Still to come: the all strategy.

Mutant outcomes

mutation_score = KILLED / (KILLED + SURVIVED)

Trend tracking across runs

Each cerebrum run is a "run" in the trend sense: its mutant records land under .cerebrum/runs/<run_id>/mutants.jsonl, and a summary row (score, kill/survive counts, the git commit at the time, the average severity of that run's survivors, and the module's code-coverage percentage) is appended to .cerebrum/history.sqlite. A per-file coverage rollup for the run — covered vs. instrumented line counts, coverage fraction, the count of mutants actually applied and tested on that file, and the count and worst severity of survivors per file — is also written to .cerebrum/runs/<run_id>/coverage.json, so coverage can be trended over time and low-coverage files ranked by risk. Every mutant now carries a Claude-estimated severity (low/medium/high/critical) alongside its mutation_type, so a declining score isn't the only signal — you can also see whether the impact of what's surviving is trending up or down over time, not just the count. cerebrum report --trend prints the last N runs; cerebrum report --survivors prints the current run's survivors (file:line, diff, severity, how many consecutive runs it's persisted) with an LLM-suggested test for each. cerebrum report --hunk-positions prints the current run's BUILD_ERROR mutants whose diff hunk header claims the wrong starting line for its own context — a distinct failure mode from a hunk header's line counts being wrong (which git apply --recount already corrects). cerebrum mutate (a one-off manual mutation) is not part of this — it still writes to the legacy flat .cerebrum/mutants.jsonl with no run or history entry. Target repos should add .cerebrum/ to their own .gitignore.