stoppower

Size your early-stopping window by statistical power instead of by habit.

Léeme en español

pip install stoppower

The problem

Most people decide a model "converged" by comparing two or three evaluations and loosening the tolerance when detection fails. That tunes the wrong knob. In the measured case (Zenodo 10.5281/zenodo.21630279), the total detection gain decomposes into 7.0× from lengthening the window and only 1.31× from changing the statistic. Tolerance barely moves the needle.

Worse: once the signal-to-noise ratio drops below 1, no tolerance recovers the decision. Type I and Type II error can no longer be separated by any choice of threshold.

The rule

For n equally spaced evaluations, the standard error of the OLS slope decreases as n^(-3/2), against n^(-1/2) for plain averaging:

SE(b) = σ · sqrt( 12 / (n(n² − 1)) )
b*   ≥ (z_α + z_β) · SE(b)

Usage

from stoppower import window_for, sigma_from_pilot, evaluate, prereg_text, pts

# 1) What window do I need so I don't miss 1 accuracy point per 2500 steps?
plan = window_for(b_star=pts(1.0), sigma=0.0082, eval_every=500)
print(plan)      # -> n=11 evaluations = 5000 steps

# 2) How much power does the window I already use actually have?
from stoppower import power_of
power_of(pts(1.0), sigma=0.0082, n=3, eval_every=500)     # -> 0.097

# 3) Decide with data in hand
evaluate(steps, accs, b_star=pts(1.0))

stoppower design --b-star 0.01 --sigma 0.0082 --eval-every 500 --prereg
stoppower power  --b-star 0.01 --sigma 0.0082 --n 3 --eval-every 500
stoppower check  history.csv --b-star 0.01 --span 5000 10000

A real case, end to end

Validation history of a model trained for 10,000 steps, evaluated every 500. The question is whether the tail is still improving or it is safe to stop.

$ stoppower check history.csv --b-star 0.01 --span 5000 10000
  sigma estimated over the 5000-10000 span: 0.010593
UNDECIDABLE · slope +0.006424 per 2500 steps (95% CI [-0.003474, +0.01632])
  t=1.27 p=0.1017 · n=11 evaluations · power=0.63 for b*=0.01
  ! power 0.63 < 0.8: this window cannot support a convergence claim,
    only "no improvement was detected". Lengthen the window before concluding.

A two-point criterion would have said "converged". Here the verdict is undecidable, with the number next to it: the window has no power to sustain that claim.

And if you hand it the whole run instead of the stable regime, it warns before answering:

$ stoppower check history.csv --b-star 0.01 --span 500 10000
  ! noise changes 7.4x within the window (sigma=0.06951 in the 1st half vs
    0.009344 in the 2nd, higher at the start): the equation assumes constant noise.
    Shorten the window or move it to the stable regime.

Three things that set it apart from copying the formula

1. σ is estimated over a span, not over the whole run. Validation noise is not constant. Measured across 8 seeds of the same model:

A factor of 9. Feeding a global σ into the equation is off by a large factor, which is why the span is a required argument here. homoscedasticity_check warns when the constant-noise assumption does not even hold inside the window.

2. It does not confuse "no improvement detected" with "converged". evaluate returns converged=None when power is insufficient, instead of a false green light. That asymmetry is the whole point: failing to reject improvement is not evidence of convergence unless the study had the power to see it.

3. It emits the pre-registerable paragraph. prereg_text(plan) produces the text with σ, its provenance, the level, the power and the resulting window — to paste into your protocol before looking at the data.

Estimating σ without circularity

Fixing a window from a pilot is not the same as recalibrating a tolerance after seeing results: what gets fixed is the design, not the verdict.

Validation

The acceptance test reproduces Table 2 of the paper (σ = 0.00821, α = .05 one-sided, power = .80): n=4 → b* = 4.56, n=8 → 1.58, n=16 → 0.55 accuracy points per 2500 steps.

Beyond arithmetic, the nominal power was checked against a Monte Carlo of 4,000 simulations per case: empirical power 0.824 against 0.80 nominal, empirical α 0.054 against 0.05, and 0.092 for n=3 against the 0.097 the formula predicts. The rule is calibrated, not just implemented.

Scope

No dependencies (standard library only). It does not train, does not touch your loop, does not decide for you. It does not model hardware effects: switching backends was measured to contribute 0.53× the across-seed variation in the late regime, negligible against what is already reported.

Citation

Speranza, M. R. (2026). Stopping criteria below the signal-to-noise floor: window length, not tolerance, governs convergence detection in architecture comparisons. 10.5281/zenodo.21630279

MIT.