evalproof

Treat LLM evaluation as measurement. Calibrate the judge, correct its pass rate, and gate CI on regressions that are real rather than noise.

Try it in the browser: evalproof.streamlit.app. Check whether a drop between two runs is real, or replay the run below and read the twelve answers that changed. It runs this repository's code; after a quiet spell it takes about half a minute to wake.

An LLM judge is a classifier with error rates. If it passes bad answers at rate FPR and good answers at rate TPR, its pass rate converges to FPR + (TPR − FPR) · p, not to the true rate p. That is classify-and-count, and it is biased the way any classifier's count is — the effect is worked through with a seeded simulation in The classifier was 95% accurate. The trend it counted was a third too flat (lab).

evalproof:

1. measures the judge against a small gold-labelled sample — confusion matrix, TPR, FPR, Cohen's κ;
2. corrects its pass rate with adjusted classify-and-count and prediction-powered inference, each with a confidence interval;
3. compares runs on paired cases — exact McNemar for pass/fail, paired bootstrap for scores;
4. gates CI only when a drop is both larger than your tolerance and statistically significant, plus latency and cost budgets, with distinct exit codes.

It is local-first: Ollama is the default provider, Anthropic and OpenAI are optional, and every model call is cached by content so re-runs are free and reproducible.

Install

pip install evalproof          # or: uv add evalproof
uv tool install evalproof      # just the CLI

Python 3.11 to 3.14. compare, calibrate and report work on run files and need no model. run needs a system to evaluate: a Python function, an HTTP endpoint, a local Ollama model, or an Anthropic or OpenAI key.

What it found on a real run

A local qwen3:1.7b support assistant for a fictional product, judged by qwen3:8b, 83 questions, one Apple M5. Full method and every table in MEASUREMENTS.md.

Five-minute quickstart

The offline example lives in the repository:

git clone https://github.com/technobasant/evalproof && cd evalproof
uv sync

# Offline: no model, no network. Four versions of a ticket router.
for v in baseline regressed noisy refactored; do
  uv run evalproof run examples/deterministic/suite.yaml \
    --system "python:router:${v}_route" --out "runs/$v.json" --quiet
done
uv run evalproof compare runs/baseline.json runs/regressed.json \
  --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 1: real regression
uv run evalproof compare runs/baseline.json runs/noisy.json \
  --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 0: 2 of 60 is noise

With Ollama installed, the LLM example is examples/support-qa (ollama pull qwen3:1.7b and ollama pull qwen3:8b; about 6–10 minutes per run on an Apple M5).

The browser app in app/ runs locally too: uv run --with-requirements app/requirements.txt streamlit run app/streamlit_app.py.

Concepts

Suite — one YAML file: a JSONL golden set, the system under test, scorers, an optional judge, calibration settings and a gate.

name: support-qa
dataset: cases.jsonl              # {"id", "input", "reference", ...extra fields}
system:
  target: ollama:qwen3:1.7b       # or anthropic:…, openai:…, python:module:func, https://…
  system_prompt_file: prompt.md
  prompt_template: "{question}"
  params: {temperature: 0, seed: 0, think: false}
judge:
  target: ollama:qwen3:8b
scorers:
  - {name: answer_key, type: regex, params: {any_of_field: accept, none_of_field: reject}}
  - {name: judge, type: llm_judge, params: {rubric_file: rubric.md}}
calibration: {judge_scorer: judge, gold_scorer: answer_key, gold_sample: 30}
gate: {metric: judge, max_drop: 0.03, alpha: 0.05, min_cases: 40, max_p95_latency_ms: 2500}

Run — evalproof run suite.yaml writes one self-describing JSON file: every case's output, latency, tokens and scores, plus the dataset hash, seed, Ollama version and model digests.

Scorers — exact_match, contains, regex (accept/reject answer keys), json_schema, numeric_tolerance, recall_at_k, mrr, ndcg, and llm_judge: a strict JSON verdict, one retry on an unparseable reply, and an explicit undecided state that is counted, never scored as a pass. evalproof pairwise judges two runs head to head in both orders and reports order-dependent verdicts as ties.

Calibrate — evalproof calibrate run.json --judge-scorer judge --labels gold.jsonl (human labels) or --gold-scorer answer_key --gold-sample 30:

Compare and gate — evalproof compare baseline.json candidate.json --suite suite.yaml --gate

From Python

The CLI is a thin layer over the same functions:

from evalproof import GateConfig, calibrate, compare_runs, decide, load_run

baseline = load_run("runs/baseline.json")
candidate = load_run("runs/candidate.json")

cmp = compare_runs(baseline, candidate)  # paired cases: exact McNemar, bootstrap intervals
decision = decide(cmp, GateConfig(metric="judge", max_drop=0.03))
print(decision.status, decision.exit_code, decision.warnings)

report = calibrate(baseline, "judge", gold_scorer="answer_key", gold_sample=30)
for est in report.estimates:  # raw judge rate, gold only, ACC, PPI
    print(est.method, est.point, est.low, est.high)

The names in evalproof.__all__ are the supported API. Other modules are internal and may change between minor versions; see the changelog.

Architecture

The architecture diagram is in the README on GitHub.

Decisions

In CI

The composite action runs compare --gate, appends the report to the job summary, and fails the job on exit codes 1, 3 or 4:

- uses: technobasant/evalproof@v0.2.0
  with:
    baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
    candidate: eval/candidate.json   # produced on this branch
    suite: eval/suite.yaml

This repository's own CI runs the offline example on every commit and asserts that the gate returns 1 for the real regression and 0 for the noisy and refactored candidates.

How it relates to other tools

promptfoo (open-source, local evals of prompts, models and RAG pipelines with deterministic and model-graded assertions, a GitHub Action, and red teaming), DeepEval (pytest-style LLM tests with research-backed metrics such as G-Eval), Ragas (LLM-driven metrics and experiments for LLM and RAG applications) and Inspect (the UK AI Security Institute's framework for model and agent evaluations, with sandboxed tools and a log viewer) are broad evaluation frameworks.

evalproof is deliberately narrow: it is the measurement layer — how much a judge's verdict is worth, and whether a difference between two runs is real. Its run file is plain, documented JSON, so it can sit beside an existing harness: produce the per-case verdicts however you like, then calibrate and gate here.

Limits

• The measurements cover one machine, two small local models, one fictional domain and 83 cases. They show the method; they are not a benchmark of those models.
• Calibration is only as good as its gold labels — MEASUREMENTS §4 shows a regex answer key being wrong more often than the judge. Use human labels when the decision matters.
• ACC needs both classes in the gold sample and a judge better than chance; PPI needs the gold sample to be random. evalproof reports when either does not hold rather than printing a number.
• The percentile-bootstrap p-value for continuous scores is an approximation; pass/fail metrics use the exact McNemar test.
• Cost tracking needs a price table in the suite for hosted models.

Roadmap

• Stratified gold sampling (by tag) and per-tag calibration.
• Multiple-comparison correction when more than one metric is gated.
• A reasoning judge (think: true) measured against the same answer key.
• Importers for other harnesses' per-case results.

Project

Changelog · Contributing · Security policy · Code of conduct · How to cite. Releases are published to PyPI from GitHub Actions with trusted publishing, and each file carries a signed provenance attestation.

Licence

Apache-2.0. See LICENSE and NOTICE.