athena-eval

Executable A/B validity gates. The arithmetic half of "is this result allowed to decide anything?", as code that can fail instead of prose in a style guide.

It exists because written-down rules did not stop the mistakes they described. Judgement about what to measure stays human; whether a measured gap is allowed to be called a winner is arithmetic, and arithmetic should not be re-derived by hand each time — least of all by a party motivated to find an effect.

What it refuses to let ship

from athena_eval import verdict, assert_inputs

assert_inputs(prompt=prompt, corpus=corpus)      # prints byte counts, refuses empties

v = verdict("baseline", a_scores, "challenger", b_scores, higher_is_better=True)
v.kind      # "decided" | "cannot-separate" | "VOID"
v.winner    # the named arm, or None
print(v)    # decided -> challenger: gap 0.010 > spread 0.000 (higher is better). ...

State the metric direction. higher_is_better has no safe default: a gate that guesses will confidently name the slower provider the latency winner.

Below n=10 a verdict is VOID, not weak evidence. For paired comparisons, fewer than 6 discordant pairs cannot reach p<0.05 at any split, however lopsided the totals look.

Install

pip install athena-eval                  # gates only, no dependencies
pip install "athena-eval[llm]"           # adds complete_checked, via athena-llm

complete_checked imports athena-llm lazily inside the function, so the gates themselves pull in nothing.