evalreliability

Measure the reliability of LLM-as-judge evaluations.

Your eval score went from 3.80 to 4.10 after a prompt change. Did anything improve?

Run the same judge on the same items twice and you may find it moves 0.68 on its own. Then 0.30 was never a result. Most teams have never measured this, because no tool in the eval stack reports it.

Other tools give you a judge score. This one gives you the error bar on that score, and tells you which decisions it can carry.

Install

pip install evalreliability

Core depends only on numpy, pandas and scipy. Harness integrations are extras:

pip install "evalreliability[inspect]"

Status

Early. 0.0.1 ships the input contract and design checking. The measurement verbs land next — see the roadmap below.

Input

One long frame, one row per rating:

item, rater and score are required. rater_type, run and dimension are optional, and each one you supply unlocks another facet of the analysis.

Design checking

Every reliability coefficient assumes a measurement design. Choose the wrong one and the number is wrong in a way nothing downstream will flag. So the design question gets answered from the data, first:

import pandas as pd
from evalreliability import check_design

df = pd.read_parquet("ratings.parquet")

print(check_design(df, by="dimension")["coherence"])

coherence: crossed, 1600 items x 3 raters, 3 ratings per item

When a two-way model is not licensed, it says so and why:

relevance: nested, 400 items x 12 raters, 3 ratings per item
  two-way models are not licensed here (raters differ across items);
  use a one-way model

And it warns where precision will be poor regardless of how many items you have:

coherence: crossed, 1600 items x 3 raters, 3 ratings per item
  warning: 3 raters gives 2 df on the rater term, so absolute-agreement
  intervals will be very wide

That last one matters more than it looks. Absolute-agreement reliability depends on a rater variance component estimated from the number of raters, not the number of items. Sixteen hundred items do nothing for it.

Roadmap

What this does not do

It will not tell you whether a judge is correct. No tool can. There is no ground truth for whether a summary is coherent — only what qualified people think, and they disagree. In SummEval, three expert annotators reached a Krippendorff's alpha of 0.41 before reconciliation.

So "correct" is not available as a target. What is available is how far a judge sits from a human reference whose own noise you have measured, and how much of your reported number is signal.

Design principles

• Correct over comprehensive. Every coefficient is tested against an independent implementation — pingouin, krippendorff, GeneralizIT — rather than against itself.
• The form follows the decision. You say whether the score value carries your decision or only the ordering. The library derives which coefficient that implies and names it in the output. No dropdown of six options you have to already understand.
• Small install. A measurement library nobody installs measures nothing.
• Say what is not supported. Where a claim is not licensed by the data, the output says so rather than printing a number anyway.

Citing

See CITATION.cff. Each release is archived on Zenodo with a DOI.

License

Apache-2.0. See LICENSE and NOTICE.