---

What is EvalFloor?

When you try k variants against an eval set and keep the best score, that score is biased upward. The maximum of k noisy measurements exceeds the true value even when all k variants are equally good, and the bias grows with k.

EvalFloor computes that bias from the scores your tuning loop already produced. No model, no rerun, no dependencies.

+7.0 points free	200 eval examples, 30 variants tried, no real difference between any of them
2 functions	check() for the floor, confirm() for a paired held-out test
0 dependencies	no model, no API key, no rerun. It reads scores you already have
Catches both errors	when a gain is fake, and when your split is too small to say

Quickstart

pip install git+https://github.com/novaleolin/evalfloor.git

(PyPI release pending. pip install evalfloor once it lands.)

You tried 30 prompts and kept the best one. The score went 0.62 to 0.69.

import evalfloor
print(evalfloor.check(scores=my_30_scores, n_examples=200))

  baseline              0.620
  best                  0.685   apparent gain +0.065
  selection floor       +0.069   <- a 30-candidate search scores this much on pure noise
  best, de-biased       0.616

  BELOW THE FLOOR -- this search has not shown anything

All 30 prompts in this example have the same true accuracy. The 6.5 point gain is sampling error, and it falls under the floor.

python3 examples/quickstart.py     # 30 seconds, no downloads, no API key

Runs two tuning sessions. In the first, all 30 variants have the same true accuracy. In the second, one variant is 8 points better. Both report a gain.

Why this happens

Each score is an estimate measured on a finite eval set, so each carries sampling error. Taking the maximum selects for positive error. Trying more variants raises the expected maximum.

eval set	5 tries	10 tries	30 tries	100 tries
50	+8.0	+10.4	+13.9	+16.8
100	+5.6	+7.5	+10.0	+12.1
200	+4.1	+5.3	+7.0	+8.6
500	+2.6	+3.4	+4.5	+5.5
2000	+1.3	+1.7	+2.2	+2.7

At 200 examples and 30 variants the floor is +7.0 points.

Usage

check: how much of your best score is luck

evalfloor.check(scores, n_examples)          # scores = every variant you tried

scores must contain every variant you evaluated, not just the winner. The floor depends on how many were tried.

confirm: does the winner hold up on data it wasn't chosen on

evalfloor.confirm(baseline_correct, winner_correct)   # per-example, True/False

  held-out   13 fixed / 3 broken   sign test p=0.0213
  CONFIRMED -- the winner is better on data it was not selected on

It also tells you when you simply don't have enough data:

  held-out   5 fixed / 0 broken   sign test p=0.0625
  UNDERPOWERED -- 5 disagreements can never reach p<0.05, no matter how
                  one-sided. Your held-out split is too small. Add examples.

An exact sign test over d disagreements cannot return a p-value below 2^(1-d), so five or fewer can never reach 0.05. confirm() reports this as UNDERPOWERED rather than as a negative result.

staged_floor: for staged evaluation

Score every candidate on a small set, promote the survivors to a larger one, report the best:

evalfloor.staged_floor(stages=[(10, 0.0), (60, 0.40), (200, None)],
                       k=30, p=0.20, nested=True)

  selection floor       +0.110

  the reported best came from:
    stage 2:   60 examples, promote above 40%      100%
    stage 3:  200 examples                           0%

  Your headline number is coming from the 60-example stage 100% of the time.
  That is not the 200-example stage you pay for.

With a 40% gate and candidates at 20%, few candidates reach stage 3. The reported maximum comes from stage 2, so the floor is +0.110 rather than the +0.059 of a 200-example stage. A stricter gate raises the floor.

API

from evalfloor import check, confirm, selection_floor, staged_floor, eb_shrink

check(scores, n_examples, baseline=None)   # .apparent_gain .floor .shrunk .beats_floor
confirm(baseline_hits, new_hits)           # .wins .losses .p_value .confirmed .underpowered
selection_floor(k, n, p)                   # points a k-candidate search gets free
staged_floor(stages, k, p, nested=False)   # same, for staged loops
eb_shrink(scores, n)                       # de-biased best

Schema optimizer (optional)

pip install "evalfloor[local] @ git+https://github.com/novaleolin/evalfloor.git"
evalfloor mydata.jsonl --kind choice --metric exact

Searches over a typed decision schema: instruction text, option descriptions, which fields go into the state, and thresholds. Every run prints its own floor and held-out test. The default backend is a local model, so no API key is needed.

python3 examples/banking77_intent.py    # ticket routing
python3 examples/rag_relevance.py       # keep-or-drop retrieved passages

Ticket routing, real output:

  winner   (train)          0.521   apparent gain +0.083
  selection floor (null)    +0.141   <- the gain is BELOW the floor

  winner   (held-out)       0.542   real gain +0.208
  held-out paired           13 fixed / 3 broken   p=0.0213

  verdict: CREDIBLE

Here the train gain of +0.083 is under the floor of +0.141, so the verdict rests on the held-out split alone.

The RAG example starts at F1 = 0.000. The scorer gives every passage 0.10 to 0.19, so the default 0.5 threshold returns an empty set. The search reaches 0.286 by lowering it.

Limits

selection_floor assumes candidates are independent, each evaluated once, on a 0/1 metric. Correlated candidates and heavy-tailed metrics both make the true floor higher than it reports. Staged loops use staged_floor instead.

All of these bias it in the same direction, so the reported floor is a lower bound. A gain under it is under the true floor too. A gain over it still needs confirm().

FAQ

I tuned my prompt 30 times and accuracy went up 5 points. Is that real? Probably not. At 200 eval examples the floor for 30 variants is +7.0. Run check() on all 30 scores to get the floor for your own numbers.

How is this different from a held-out set? A held-out set establishes that a variant is better. The floor tells you when your scores cannot establish anything, before you spend held-out data on them. Run check() first and confirm() on whatever survives.

Is this just overfitting to the eval set? Nothing is fitted here. The bias comes from picking the largest of several noisy scores, which is biased upward whether or not a model was trained.

My metric isn't accuracy. selection_floor assumes a 0/1 metric. On other metrics it under-reports the floor, so a gain under the reported floor is under the true one as well.

My loop promotes candidates between cheap and expensive stages. Use staged_floor(). It also reports which stage the maximum came from, which is usually not the expensive one.

Can I use it on hyperparameter sweeps, A/B tests, model selection? Yes. Anything that evaluates k options and keeps the best has this bias.

MIT.