Harness Overfit Detector (HOD)

Detect when an agent-harness improvement is real and when it is just fitted to the eval suite.

Every harness improvement is usually measured on the same fixed set of tasks. A change tuned to those tasks can raise the score without improving anything in general, and nothing detects it. HOD adds a second lens: tasks the change was never tuned for, and the gap between the two measurements.

Installation

From the repo checkout (no install needed, run from the repo root):

python3 -m hod --help

Editable install from a checkout, for the hod command:

pip install -e .

Usage

The hod command has three measurement sections, a robustness layer and an automation layer (including the improve self-improvement loop). From a clone without installing, replace hod with python3 -m hod in every command below.

generate: held-out tasks

Generates new tasks in catalog format (setup, instruction, expected behavior, auto-fail, 0–2 rubric) from 8 parametric families, reproducibly by seed. Each family is a set of setup and instruction templates times parameter axes (files, values, requests, material differences), so the space is 13,272 combinations instead of the 18 fixed variants of the initial design. Every candidate task passes three gates before it is emitted:

• validity: no empty field, no unfilled placeholder, complete 0/1/2 rubric;
• similarity against the fixed suite (max of Jaccard and containment on 5-grams, ceiling 0.30);
• novelty: setup + instruction compared with every task already in the set and with an exclusion buffer (--exclude DIR, sets already used), Jaccard on 3-grams, ceiling 0.60.

If no candidate passes, generation fails closed instead of emitting a contaminated or duplicated task. Under the novelty gate the space is estimated at roughly 1,500 mutually distinct tasks (about 70 to 800 per family). With --seeds, each seed's set excludes the previous seeds' tasks, so multi-seed sets are disjoint.

# Generate 8 held-out tasks (seed 42 reproduces the same set)
hod generate --seed 42 --n 8 --out runs/heldout-42

# Against a different fixed suite
hod generate --seed 42 --n 8 --out runs/heldout-42 --catalog path/to/catalog

# A fresh set that shares nothing with sets already used
hod generate --seed 43 --n 40 --out runs/heldout-43 --exclude runs/heldout-42

# Frontier sets (Ornith-1.5 style): families where the baseline succeeds
# near the target rate get more tasks; saturated families keep a floor
hod calibrate --heldout runs/heldout-42 --baseline held-base.json --frontier-out frontier.json
hod generate --seed 44 --n 40 --out runs/heldout-44 --frontier frontier.json --frontier-target 0.35

With weights (--family-weights or --frontier), each family gets a quota proportional to its weight (largest remainder), so the set stays stratified.

The output directory holds one .md file per task plus manifest.json (id, family, file, sha256, max similarity).

gap: suite vs held-out

From suite and held-out scores before and after a change, reports means, deltas, the gap (suite minus held-out), 95% confidence intervals from a paired bootstrap, the placebo noise floor, coverage, power (minimum detectable effect and the held-out size you would need) and a verdict. Score files are JSON maps of {task_id: 0|1|2}. --decision point reproduces the 0.1 point rule; --strict asks for more evidence before GENUINE.

hod gap --suite-base scores/suite-base.json --suite-cand scores/suite-cand.json \
  --held-base scores/held-base.json --held-cand scores/held-cand.json --out gap-report.md

audit: contamination check

Verifies manifest and hashes, cross-family near-duplicates, duplicate tasks (setup + instruction) and held-out similarity against the suite.

hod audit --heldout runs/heldout-42 [--catalog path/to/catalog]

Robustness

Beyond the three measurement sections, HOD has a robustness layer. Every layer is optional except through hod pipeline, which runs all of them.

prereg: hypothesis first

The change author declares what they expect before seeing the held-out set. The pipeline refuses to measure if the prereg file is newer than the generated sets (peeking). After the gap, the prediction is checked against the observation.

hod prereg --change "new routing rule" --expect-held up --min-ratio 0.75 --out prereg.json
hod gap ... --prereg prereg.json

calibrate: difficulty and Q

Baseline scores on the candidate tasks reveal which tasks discriminate and which are saturated. Difficulty is (2 - baseline) / 2, novelty is 1 - max similarity, and Q is validity x difficulty x novelty. Saturated tasks (baseline 2) are filtered out; if all saturate, the report says NO SIGNAL instead of measuring a gap on noise.

hod calibrate --heldout runs/heldout-42 --baseline held-base.json \
  --out runs/heldout-42-cal --report runs/cal-42.md

placebo: empirical noise

A placebo run is a pipeline run with a no-effect change. Its gap is pure noise. hod placebo aggregates placebo gaps from the ledger into a calibration file, and hod gap --calibration flags whether an effect is bigger than any observed no-op effect.

hod placebo --ledger runs/ledger.jsonl --out calibration.json
hod gap ... --calibration calibration.json

ledger: the feedback loop

Every pipeline run and every canary report appends one JSONL line to the ledger (change, verdicts, gap, hypothesis, per-category verdicts). hod ledger reports totals, overfit rate and family weights: weight is 1 + the family's overfit rate, so the next generation looks harder where overfitting appeared most. hod generate --family-weights weights.json uses them; same seed plus same weights reproduces the same set.

hod ledger --ledger runs/ledger.jsonl --weights-out weights.json

sanity: does the evaluator separate good from bad?

Before trusting any gap, score a known-good and a known-bad harness on the same tasks. hod sanity passes only if good − bad has a 95% CI above zero (and, with --manifest, if no family has good ≤ bad). An evaluator that accepts both versions measures nothing.

hod sanity --good scores-good.json --bad scores-bad.json --manifest runs/heldout-42/manifest.json

agree: second rater and retest

hod agree compares two score files (second rater, or the same set scored later) and reports percent equal, mean absolute difference and Cohen's kappa. The mechanism is ready; the second rater's actual scores are future data.

hod agree --a scores-rater1.json --b scores-rater2.json

blind: anonymous scoring

hod blind replaces task ids with shuffled t01..tNN ids and writes the key elsewhere; hod unblind restores real ids after scoring. Honest limit: with a single agent, the key exists on disk and the blinding is instruction-enforced (do not open the key); with a second rater who never sees the key, it is real blinding.

hod blind --heldout runs/heldout-42 --template run/heldout-blind.json --key keys/heldout-key.json
hod unblind --scores run/heldout-blind.json --key keys/heldout-key.json --out run/heldout.json

Automation

One command runs all layers end to end: prereg timing check, wave 1 gate, multi-seed gaps, hypothesis check, placebo significance, ledger append, report. Exit codes: 0 completed, 3 stopped at wave 1 (OVERFIT, no expansion), 1 validation failure.

hod pipeline --change "new routing rule" --prereg prereg.json \
  --heldout-dir runs/heldout --scores-dir runs/scores \
  --seeds 42,43,44 --ledger runs/ledger.jsonl --out runs/pipe-01

Because multi-seed sets are now disjoint, the pipeline also pools all held-out tasks into one bootstrap and reports the pooled CI verdict (pooled in summary.json).

improve: self-improvement loop

hod improve runs a proposer (your command) that suggests harness changes and accepts only the ones HOD calls GENUINE. Each round: it checks that the evaluator has not changed (hash), asks the proposer for k candidates (the proposer sees the champion, its public suite scores and aggregate history, never held-out tasks), generates a fresh held-out set that excludes every task used before, measures the champion and a placebo rerun of it (noise floor), optionally checks evaluator sanity, then runs each candidate through the gates in order: your --validate command, coverage, CI verdict = GENUINE, no family regression on the held-out or the suite. The champion is replaced (atomic write) only after all gates pass. A sealed final-exam set, generated before round 1 and excluded from every round, compares the final champion with the initial harness at the end. Everything goes to lineage.jsonl: each candidate, the gate that stopped it, its numbers and a lesson with its invalidation trigger.

With --frontier-target 0.35, each round's held-out set leans toward the families where the champion's public suite success is near the target (the sealed final exam stays uniform).

Stop rules: --rounds, --budget (candidate evaluations), --patience (rounds without an acceptance), an empty proposal, held-out exhausted, failed sanity, changed evaluator.

hod improve --initial harness.json \
  --eval "my-eval --config {config} --tasks {tasks} --seed {seed} --out {out}" \
  --propose "my-proposer --champion {champion} --history {history} --k {k} --round {round} --out {out}" \
  --catalog path/to/catalog --out runs/rsi-01 \
  --sanity-good good.json --sanity-bad bad.json --validate "my-check {config}"

{tasks} is a JSON list of {"id", "family", "text"}; the evaluator writes {task_id: 0|1|2}; the proposer writes a JSON list of candidate configs (each may carry "change", a description). Exit codes: 0 something accepted, 2 nothing accepted, 1 error.

demo: offline proof

hod demo --out runs/demo [--sweep 30] [--via-commands] runs improve against a deterministic toy harness (hod/toy.py: per-family skill knobs that transfer, a "memorize the suite" knob that does not, a cosmetic knob) with a scripted proposer that offers a genuine, an overfit, a placebo, a trade-off (one family zeroed) and a mixed change every round. No network, no model.

Hooks, not cron and not a daemon: automation is event-triggered. The canary run generates multi-seed sets, blind templates and the frozen prereg copy; the canary report computes per-seed and per-category gaps, calibrates against ledger placebos and appends to the ledger by itself. Nothing polls, nothing runs on a schedule.

How the verdict works

Default (--decision ci). "Up" means the 95% CI is above zero and the point estimate is above the placebo noise floor; "down" is the mirror image. t50 = held − ½·suite and t75 = held − ¾·suite test the transfer as linear combinations, which stay stable when the suite gain is small.

Coverage below 90% (tasks without a score) makes the verdict INCONCLUSIVE: a missing score is "not run", never "passed". The 0.1 point rule is still in every report as verdict_point:

With a small held-out sample (under 8 tasks) the point verdict is flagged as indicative, and the CI verdict warns below 30 tasks and estimates the size you would need.

Held-out tasks must never enter the fixed suite: once they do, they stop being held-out. The candidate change must be frozen before the held-out set is generated; if it changes after, the run is invalid.

Canary integration

The harness canary generates a held-out set on every run and reports the held-out table, gap and verdict next to the suite table. The verdict is advisory by default; HELDOUT_GATE=blocking makes OVERFIT (aggregate) block. Old runs without held-out data still report normally, and a failed generation or verdict computation degrades softly instead of breaking the suite gate. The canary uses the repo checkout directly (PYTHONPATH=<repo> python3 -m hod generate), so it works without a pip install. With HOD_SEEDS="42,43,44" it generates one set per seed and reports the aggregate verdict and stability; with HOD_BLIND=1 it writes blind templates and keeps keys outside the run dirs; with a prereg file argument it freezes the hypothesis copy in the run dirs. The report appends to runs/ledger.jsonl by itself unless HOD_NO_LEDGER=1.

Layout

hod/         the package: cli, generate, gap, stats, audit, families,
             prereg, calibrate, placebo, ledger, agree, blind, sanity,
             pipeline, improve, toy, demo
tests/       synthetic positive controls + CLI smoke test + robustness +
             pipeline e2e + audit regressions + stats, generator, improve
runs/        example held-out set, first real baseline and control fixture

License

Apache 2.0. See LICENSE.