homefield

Benchmarks are away games. Test models on your home field.

A new model drops. The leaderboard says it's 6 points better. Your timeline says it's a game changer.

You switch. And on your codebase, with its weird build, its legacy auth module and its tests that need a running Redis, it's… fine? Worse? You honestly can't tell, because you're comparing vibes.

Public benchmarks measure public repos. Models have probably seen those repos. None of them have seen yours.

homefield turns your own git history into a private benchmark. Every past commit where someone changed code and added a test becomes a task: here's the commit message, make the change. The tests from that commit, hidden until the end, decide who passed.

A real run: Haiku vs Sonnet on simonw/llm

I mined simonw/llm's 2026 history: 16 real tasks, each a commit that added a test the old code fails. 6 more candidates were dropped (flaky, or needing dependency changes). Then I ran two Claude models, one attempt per task:

$ homefield mine --since 2026-02-01 --stability-runs 2
Mined 16 task(s) into .homefield/tasks.jsonl
Dropped: reference fails 5, tests pass without the change 1

$ homefield run --agent claude:haiku --agent claude:sonnet --attempts 1 --budget 1.0
$ homefield report

- claude:haiku vs claude:sonnet: no detectable difference (Δ=+0%, 95% CI -19%…+19%, p=1) · McNemar exact over 16 tasks

The "cheap" model wasn't cheap. It tied on solve rate, but it took almost 5× longer per attempt and cost almost 4× more per solved task. That's the kind of thing you only learn on your own code.

Caveats, stated plainly: 16 tasks is small (the interval on the difference is ±19 points), there was one attempt each, and agents ran edit-only, so they couldn't run the tests themselves, which is part of why fewer than half were solved. Every task, attempt and cost is in examples/simonw-llm/.

New in v2: is it better, or did you get lucky?

From the real run above:

## Head to head
- `claude:haiku` vs `claude:sonnet`: no detectable difference (Δ=+0%, 95% CI -19%…+19%, p=1) · McNemar exact over 16 tasks

Every comparison now comes with a paired test, a confidence interval and a plain-English verdict. With one attempt per task it's an exact McNemar test on which tasks each agent solved. With several attempts (the default is 3) it's a paired permutation test that treats tasks as the unit. The report names the test it used. I measured why that matters by simulating 500 comparisons of two identical agents on 30 tasks:

And the uncomfortable flip side: a real 15-point gap (75% vs 60%) on 30 tasks × 3 attempts is detected only half the time. Most internal evals are too small to see the differences people argue about. homefield tells you when yours is.

What else v2 adds:

• Flaky tasks are thrown out. Each task's controls run 3× (--stability-runs). A test that passes sometimes never becomes a task, and the report says how many were dropped.
• Multiple attempts, honest statistics. 3 attempts per task by default, unbiased pass@1 / pass@k, and comparisons that treat tasks as the unit, because attempts on the same task are correlated and counting them separately overstates confidence.
• Vague tasks are flagged. A commit message like "fix" can't tell any model what to do. Tasks are scored for how well-specified they are, and results are also reported on the well-specified ones only.
• A memorization check. homefield probe asks the model to reproduce each commit's code from the message alone, with no repo access. Tasks it can recite are flagged, and --exclude-memorized skips them.
• Resumable. --resume picks up a crashed or interrupted run without redoing finished attempts.

Which rules in your CLAUDE.md actually do anything?

Everyone's CLAUDE.md grows. Nobody knows which lines matter. So measure it (illustrative output):

$ homefield ablate CLAUDE.md --agent claude:sonnet --attempts 3
14 rule(s) in CLAUDE.md, 20 task(s), 1 agent(s), 3 attempt(s) each, 15 variants
Plan: 900 agent runs, ~22.5 hours sequential, ~$270.00
Nothing run. Re-run with --yes to start.

It prints the plan and cost first, and nothing runs until you add --yes. Then it removes one rule at a time, reruns your tasks, and labels each rule load-bearing (removing it hurts), harmful (removing it helps), or no detectable effect, with Holm-corrected p-values across rules and the ~token cost each rule adds to every single turn. Your CLAUDE.md is never edited; the report is saved to .homefield/.

I haven't run a real ablation yet: 15 variants × 20 tasks × 3 attempts is hundreds of dollars, which is why ablate prints the plan first. The method is proven by simulation; real results will be linked here.

"No detectable effect" isn't proof a rule is useless, and the report says so. With few tasks, small effects are invisible. The tests prove the method: in a simulated 6-rule file where one rule helps and one hurts, ablation finds exactly those two.

Install

uv tool install git+https://github.com/sandeepsirodia/homefield   # or: pipx install git+https://…

One Python file, zero dependencies. You need git and whichever agents you want to pit against each other.

Why the results are trustworthy

Every task is proven solvable, and proven to need work. A commit only becomes a task if its new tests fail on the old code and pass on the real change. A task nobody could solve, or one that needs no change at all, never makes the list.

The agent can't peek at the answer. The obvious approach, a git worktree, shares your object store, so a curious agent could run git log --all and read the real fix. homefield instead gives each agent a fresh one-commit snapshot built with git archive. The future isn't hidden; it simply isn't there. A test in this repo checks exactly that.

It dodges memorization. --since keeps only commits newer than your models' training data, and homefield probe flags tasks a model can recite anyway.

Nothing leaves your machine. Tasks, hidden tests and results live in .homefield/. Only the agents' own API calls go out.

Bring any agent

# untested recipes, so check the flags against your CLI version:
--agent 'codex=cmd:codex exec --full-auto "$HOMEFIELD_PROMPT"'
--agent 'gemini=cmd:gemini -p "$HOMEFIELD_PROMPT" --yolo'
--agent 'aider=cmd:aider --yes --message "$HOMEFIELD_PROMPT"'

Safe by default

The Claude adapter is edit-only out of the box (--permission-mode acceptEdits): it can change files but not run commands. Add --allow-shell to let it run tests too. That usually raises solve rates, but only do it inside a container or VM.

Upgrading from v1

run now makes 3 attempts per task by default (it used to make 1). Pass --attempts 1 for v1 behavior. Old run files still load, and the v1 report table is unchanged, with the new sections added below it.

Prior art, and what's new here

• SWE-bench established the method: real issues from real repos, graded by the repo's own tests. homefield applies it to your repo.
• RepoTrials and commit-replay-bench also turn git history into private benchmarks.
• claude-instruction-ablation scores CLAUDE.md rules with a rubric.

What homefield adds:

• snapshots with no reachable future history
• flake-filtered tasks
• paired, task-level statistics with plain verdicts
• a memorization probe
• CLAUDE.md ablation measured by actual solve rates, with Holm correction and a cost plan up front

python -m unittest discover -s tests -v

_{MIT © Sandeep Sirodia · Ran it on your repo and got a surprising winner? Open an issue and tell me. A ⭐ helps too.}