eval-preflight

A pre-flight check for eval results. It catches partial runs, failed judges, mislabelled scores and inconsistent aggregation, so the number you gate on is the number your eval actually produced.

Status: alpha (0.2.0). SPEC.md is the contract the code is built against.

The problem in ten seconds

An LLM judge scores 1 case out of 4 and raises on the other 3. pydantic-evals reports its average as 1.0, the same number a judge that scored all 4 cases would produce (pydantic/pydantic-ai#7928). A score >= 0.8 gate passes on evidence that mostly does not exist.

Two evaluators share a default name and one fails on a case: the survivor's score is filed under the failed one's name for that case (pydantic/pydantic-ai#10636). The headline Judge average mixes two different rubrics.

Neither run raised. Both produce a clean-looking table. eval-preflight reads the raw per-case results and answers one question before anyone acts on the number: can these numbers be trusted as a description of this run?

$ eval-preflight check name_shift.json
eval-preflight  name_shift · 4 cases · pydantic-evals

PASS  task-failures           0/4
FAIL  evaluator-coverage      Judge('strict') scored 1/4 case-runs (25%)
PASS  evaluator-coverage      Judge('lenient') scored 4/4 case-runs (100%)
FAIL  evaluator-errors        Judge('strict') errored on 3/4: RuntimeError: judge quota exhausted
PASS  evaluator-errors        Judge('lenient') errored on 0/4
FAIL  name-collision          "Judge" → Judge('strict') on c1; Judge('lenient') on c2, c3, c4
FAIL  name-collision          Judge('lenient') → "Judge_2" on c1; "Judge" on c2, c3, c4
PASS  aggregation-consistency Judge('strict'): pooled = mean of per-case means = 0
PASS  aggregation-consistency Judge('lenient'): pooled = mean of per-case means = 1
PASS  constant-output         no score or label evaluator returned a single value on ≥5 case-runs

Verdict: FAIL (4 failures, 0 warnings)

That file is tests/fixtures/pydantic_evals/name_shift.json, a report written by the real pydantic-evals 2.61.0 (scripts/make_fixtures.py).

Install

Python 3.11+. Runtime dependencies: pydantic and rich. No network access, no telemetry.

$ pip install eval-preflight
$ pip install "eval-preflight[pydantic-evals]"   # also the in-memory pydantic-evals path

Quickstart

CLI

$ eval-preflight check results.json                         # format auto-detected
$ eval-preflight check results.json --baseline main.json    # also compare with a baseline run
$ eval-preflight check results.json --json > preflight.json # machine-readable report

Exit codes: 0 PASS (or WARN), 1 FAIL (or WARN with --warnings-as-errors), 2 the check could not run (unreadable file, unknown format, bad flag). Every threshold is a flag; see eval-preflight check --help and docs/ci.md.

Python

from eval_preflight import check, load

result = check(load("results.json"))
result.verdict             # "PASS" | "WARN" | "FAIL"
result.raise_if_failed()   # IntegrityError listing the FAIL findings

For a live pydantic-evals report, without writing it to disk:

from eval_preflight import check_pydantic_evals

report = dataset.evaluate_sync(task)
check_pydantic_evals(report).raise_if_failed()

The live report also carries the framework's own averages, so aggregation-consistency can check that each reported number is reproducible from the raw results, and from how many cases.

CI

Run the evals, then gate on the pre-flight before anything reads the score. examples/github-action.yml is a complete workflow; this repository's CI runs its pre-flight steps against a clean run (must pass) and a partial run (must fail). Flags, baselines and the JSON schema: docs/ci.md.

What it checks

Evaluator identity comes from the framework's evaluator spec (Judge(rubric='strict')), never from the display name, because the display name is one of the things being checked. Each check, its rationale, thresholds and example output: docs/checks.md.

Why aggregation gets its own check: with repeat=2, case a passing 1 assertion and case b failing 3, the pooled assertion pass rate is 0.25 and the mean of per-case means is 0.5. pydantic-evals' report.averages() reports 0.5; eval-preflight says so and names the estimator (golden scenario C, tests/test_golden.py).

Supported inputs

The pydantic-evals adapter is tested against reports generated by running pydantic-evals 2.61.0, and the promptfoo adapter against exports generated by running promptfoo 0.125.1, not hand-written JSON. A result without its evaluator source is rejected rather than attributed by name. CI also installs the newest pydantic-evals 2.x, on every push and weekly, runs the suite against it and regenerates the fixtures, failing if a single byte changes (ci.yml, job upstream).

Relation to other eval tools

eval-preflight runs before the statistics. It says whether a run's numbers describe that run; it does not say whether model A beats model B.

• evaltrust ("an auditor for LLM evaluations") checks how much confidence a comparison between eval results deserves.
• limen-eval measures the same-configuration noise floor from repeated runs: per-item flakiness and sign stability.

They are complementary: a significance test on a mean computed from a quarter of the cases is precise about the wrong number.

Limitations

• Three input formats: generic, pydantic-evals and promptfoo (one prompt and provider per export; details). Inspect AI, deepeval, ragas and LangSmith are on the backlog.
• It checks completeness, attribution and arithmetic. It does not judge whether an evaluator or rubric is any good; constant-output is a heuristic for a stuck judge, not proof.
• Identity is only as good as the framework's evaluator spec. pydantic-evals omits the field name for a single positional argument, so the id renders positionally (Judge('strict')) and is not comparable with an id written by hand in the generic format.
• A run with two measurements under one evaluator id in a case-run, or one id with two kinds, is rejected (exit 2) rather than checked. pydantic-evals can produce both: two evaluators with the same spec, or an evaluator whose return type varies by case (input format). So is a measurement with both a value and an error: an exporter that writes score: 0.0 next to a judge failure must write value: null instead.
• JSON reports from pydantic-evals do not record the producing version or the framework's averages; only the in-memory path can check reported aggregates.
• pydantic-evals reports record evaluator results, not which evaluators were attached to which case. An evaluator scoped to some cases (Case(evaluators=...)), or one that returns {} on some cases, can therefore fail evaluator-coverage although nothing was lost. --min-coverage can silence it, but would also hide a judge that silently dropped cases.
• Alpha: the data model is stable from 0.1.0; flags and messages may still change before 1.0.

Development

$ uv sync
$ make check      # ruff, ruff format --check, mypy --strict, pytest with coverage >= 95%
$ make fixtures   # regenerate the pydantic-evals fixtures by running the pinned library

Golden scenarios A–E in SPEC.md §9 are the acceptance tests; the clean run must always pass. CI also builds the wheel, installs it in a fresh virtualenv and runs the quickstart against it (scripts/smoke_dist.sh).

License

MIT