eval-flake-detector

Detect flaky LLM eval cases across repeated runs. Computes per-case pass rate, standard deviation, and a flakiness severity ("stable" / "low" / "medium" / "high"). Pure stdlib, zero runtime dependencies.

Python port of @mukundakatta/eval-flake-detector. The JS-compatible shape is exposed as detect_eval_flakes so existing dashboards keep working.

Install

pip install eval-flake-detector

Usage

from eval_flake_detector import detect

# One inner list per case; entries are repeat results.
runs = [
    [True, True, True, True, True],            # case 0: stable pass
    [False, False, False, False, False],       # case 1: stable fail
    [True, False, True, False, True, False],   # case 2: flaky 50/50
]

report = detect(runs, flaky_low=0.1, flaky_high=0.9, min_runs=3)

report.stable          # False -- case 2 is flaky
for case in report.flakes:
    print(case.case_id, case.pass_rate, case.std_dev, case.severity)
# case-2  0.5  0.5  high

Already have a flat list of result dicts (one row per repeat)? Pass that directly -- it's auto-grouped by id:

runs = [
    {"id": "math-1",   "passed": True},
    {"id": "math-1",   "passed": False},
    {"id": "math-1",   "passed": True},
    {"id": "lookup-2", "score":  0.95},
    {"id": "lookup-2", "score":  1.00},
]
detect(runs)

Severity bands

JS-compat output

from eval_flake_detector import detect_eval_flakes

# Same shape as the JS sibling: {"stable": bool, "flakes": [{"id", "count", "stddev"}]}
detect_eval_flakes(runs, max_stddev=0.2)

See the JS sibling's README for the original design notes.