lucky

hyperfine, but for pass rates. Is it better, or did you get lucky?

You tweak a prompt. You run your eval. 6/6. Before the tweak it was 2/6. Ship it!

That really happened to me. Same prompt, same model, same eval, run twice with no changes: 2/6, then 6/6. Nothing got better. I got lucky.

# lucky's verdict on those two runs (2/6 vs 6/6):
#1 vs #2: No detectable difference (p=0.061). To detect a gap this size, run ~n=10 each.

Pass rates lie at small n, and we all read them anyway: prompt tweaks, model swaps, agent configs, "is this flaky test fixed now?". lucky runs the thing enough times and tells you what you actually know.

Use it

uvx --from git+https://github.com/sandeepsirodia/lucky lucky -n 20 './eval.sh --model a' './eval.sh --model b'

#1: ./eval.sh --model a
  17/20 passed (85%, 95% CI 64–95%)   flaky (17/20 passed)

#2: ./eval.sh --model b
  12/20 passed (60%, 95% CI 39–78%)   flaky (12/20 passed)

#1 vs #2: No detectable difference (p=0.16). To detect a gap this size, run ~n=57 each.

85% vs 60% looks like a blowout. It isn't one yet. lucky tells you how many more runs it would take to know.

Any command that exits 0 on success works: tests, evals, scripts, curl health checks, agent runs.

Is my test flaky, or did I fix it?

lucky -n 50 -j 8 'pytest tests/test_checkout.py -x -q'

  50/50 passed (100%, 95% CI 93–100%)   stable pass

"Passed 3 times in a row" means very little. 50/50 with a 95% interval of 93–100% is a claim you can put in a PR description.

Everything it does

Runs are interleaved (A, B, A, B…), so if your API gets slower at 3pm, both commands feel it equally.

Peeking is cheating, and here's the number

The tempting move: run a few, check, run a few more, check again, stop when it looks significant. That inflates false wins. I simulated two identical coins, checking for a "significant" difference:

--until-decided checks at 5 pre-declared points and splits the 5% error budget across them. Simulated false-positive rate: 1.75%, and it still catches real gaps (0.9 vs 0.4) over 90% of the time. The simulation is a test that runs in CI.

No magic, no dependencies

Every statistic is implemented in lucky.py with the standard library and checked in tests against independent references:

• the Wilson interval against published values
• Fisher's exact test against brute-force enumeration over 200 random tables
• Holm against a worked example
• the sample-size estimate (Fleiss continuity correction) against the formula, and by simulation: at the suggested n, a real gap is actually detected about 80% of the time

It's one file. Read it in a sitting.

Honest limits

• It answers "is there a detectable difference?", not "how big is the real difference?". Look at the two intervals for that.
• It assumes runs are independent. If your command caches results between runs, lucky can't know.
• --until-decided uses a simple, conservative stopping rule (Bonferroni across looks). It's safe, but it stops later than fancier designs would.

Prior art, and what's new here

• hyperfine is the inspiration for the interface. It measures time; lucky measures pass rates.
• Flaky-test rerunners (pytest-flakefinder, flaky, go test -count) rerun tests inside one framework. Use them if you live in that framework.
• Eval statistics libraries (evalstats, evalci) are Python libraries you call on results tables, and they're more complete statistically.

lucky's niche is narrow on purpose: any command, from the shell, with a verdict, plus early stopping that doesn't cheat.

python -m unittest discover -s tests -v

_{MIT © Sandeep Sirodia · If lucky just saved you from shipping a coin flip, a ⭐ helps the next person find it.}