benchgap

Measure how much of a machine-learning benchmark score survives deployment.

A model scores 0.99 on a held-out test set. What does it score on real data six months later? The gap between those two numbers is usually reported — if at all — as a single figure, which names an effect without explaining it. benchgap helps you break it into pieces you can act on, with confidence intervals that aren't quietly wrong.

pip install benchgap

Depends on numpy and nothing else.

The part most people get wrong

Confidence intervals on model evaluations are routinely several times too narrow, because the rows being resampled aren't independent:

Resample rows independently in any of those and you'll report an interval that looks convincing and isn't. benchgap takes a groups argument everywhere it computes uncertainty, and resamples whole groups.

import benchgap as bg

lo, hi = bg.cluster_bootstrap_ci(y, prob, threshold, groups=website, n_resamples=1000)

This isn't optional politeness. It's the difference between an interval that means something and one that doesn't.

Quick start

Score a model honestly

import benchgap as bg

# Pick the decision threshold on VALIDATION data, then freeze it.
# Choosing it on the data you're about to report is the most common way an
# evaluation quietly becomes optimistic.
threshold = bg.select_threshold(y_val, prob_val, objective="tss")

scores = bg.score_at(y_test, prob_test, threshold)
print(scores.tss, scores.recall, scores.false_alarm_rate)

TSS (True Skill Statistic) = recall − false-alarm rate. It is 0 for any constant forecast, which makes the zero line a real no-skill baseline. Accuracy can't do that job: when one outcome is rare, a model that always predicts the common case scores 98% and has no skill at all.

Build a chain of comparisons

A "cell" is one model scored on one dataset. Line several up so each step changes exactly one thing, and the total drop becomes attributable instead of merely observed.

cells = {
    "shuffled":    bg.evaluate_cell("shuffled", "random split — the common protocol",
                                    y1, p1, thresholds, groups=g1),
    "honest":      bg.evaluate_cell("honest", "grouped split — no leakage",
                                    y2, p2, thresholds, groups=g2),
    "operational": bg.evaluate_cell("operational", "live data, model frozen",
                                    y3, p3, thresholds, groups=g3),
}

steps = [
    ("leakage", "shuffled", "honest", "Same data; only the split changes."),
    ("drift",   "honest",   "operational", "Same protocol; later data."),
]

print(bg.attribute(cells, steps)["tss"])
# {'leakage': 0.215, 'drift': -0.077, 'total': 0.138}

A step whose endpoints are missing is skipped rather than guessed at, and total is computed from the first and last cell directly — so it stays correct even when an intermediate cell doesn't exist.

Ask whether retraining helps

result = bg.recovery(gap_tss, retrained_tss, ci)
# {'delta': -0.136, 'ci95': [-0.255, -0.009],
#  'direction': 'harms', 'significant': True}

The interval can exclude zero in either direction. A retrained model that's measurably worse is a finding, not a null — testing only for improvement would report it as "not significant" and throw it away.

What's in it

The interface is arrays in, dicts out. It never sees a DataFrame, a model object, or a file path — the three studies that drove its design disagree about all of those, and anything wider would have forced a data model onto domains that don't want one.

Where it came from

Extracted unchanged from three independent studies that had each arrived at the same code:

Those studies make a claim that only works if every number is comparable: how far a benchmark score falls in deployment depends on what kind of change the model faces. Three copies of the scoring code kept in sync by a hash check would drift eventually — and when they did, the comparison would silently stop being a comparison. Hence one package.

License

MIT.