EvalSeal

Reproducibility receipts for LLM evals: run it N times, report the score with its noise, and seal what actually ran.

The problem

An eval score from a single run is one sample of a random process. Run the same eval again and borderline items quietly flip from PASS to FAIL, especially when an LLM judge grades them. Reports also name the model you asked for, not the one that answered. EvalSeal measures the flips, records the real provenance, and seals both into a tamper-evident ledger.

Quickstart

pip install evalseal

# The recorded demo (examples + cassette) lives in the repo.
git clone https://github.com/patibandlavenkatamanideep/evalseal && cd evalseal

# Replays the committed cassette: no API key, no network.
evalseal run \
  --dataset examples/borderline_judge/dataset.jsonl \
  --target-config examples/borderline_judge/target.json \
  --scorer-config examples/borderline_judge/scorer.json \
  --n 5

Real flip rates

Recorded 2026-09-15 and committed in tests/cassettes/run.json: gemini-2.5-flash as both the target and the judge, temperature left at the provider default, 20 arguable prompts, 5 runs each. The quickstart above replays exactly this run.

Mean score: 0.92, but 5 of 20 cases did not get the same verdict every time.

Treat the k-th repeat of every case as one ordinary single-run eval, and the five "single runs" of this identical eval scored 0.90, 0.95, 0.85, 0.90 and 1.00. A single run can't tell you which of those numbers you got.

The report also flagged TEMPERATURE NOT SET for both the target and the judge, which is the reason these borderline verdicts can come out differently from run to run.

To re-record with your own key, copy .env.example to .env, add a free Google AI Studio key, and run the quickstart with EVALSEAL_RECORD=1. If the free tier rate-limits you, run the same command again later; responses already recorded are kept.

Commands

run takes --concurrency (default 4 requests in flight), --max-retries (default 5, on HTTP 429/408/5xx and connection errors, honouring Retry-After), --timeout, and --quiet. Concurrency never changes the result: each response is recorded against its own (case, repeat) slot, so a 16-worker replay is identical to a serial one.

Gating a pipeline

--fail-on decides which stability classes fail the run, and --junit-xml writes a report CI can display next to ordinary tests:

- name: Eval reproducibility
  run: |
    evalseal run --dataset evals/dataset.jsonl \
      --target-config evals/target.json --scorer-config evals/scorer.json \
      --n 5 --fail-on borderline --junit-xml junit.xml --quiet

Start with --fail-on none to observe flip rates without blocking merges, then tighten.

Stability classes are based on the flip rate, the share of a case's N verdicts that disagree with its majority: STABLE (0), BORDERLINE (≤ 20%), UNSTABLE (> 20%).

Provenance warnings show up in the report when:

• the served model differs from the requested one (for the target or the judge),
• the endpoint isn't a canonical provider host,
• temperature was left at the provider default,
• the served model or system fingerprint changed partway through the run.

Using it as a library

from evalseal import Dataset, LocalCallableTarget, RegexScorer, run_eval

record = run_eval(
    Dataset.from_jsonl("dataset.jsonl"),
    LocalCallableTarget(my_model_fn),
    RegexScorer(r"^yes"),
    n_repeats=5,
)
print(record.aggregate.mean_score, [r.stability for r in record.results])

The package ships type information (PEP 561), so mypy and pyright see the annotations.

How it works

The executor sends each prompt to the target N times and scores every response. An LLM judge is itself a target, so its own randomness is measured instead of assumed away. analyze.py computes the mean, a seeded bootstrap 95% CI, and the flip rate for each case. Every request goes through a cassette, keyed by the request plus which repeat it belongs to. In record mode real responses are saved as they arrive; in replay mode, the default and what CI uses, they are served back by that key, and a missing entry fails loudly. Each run is saved as a RunRecord: its manifest (requested vs. served model, fingerprint, parameters and whether they were set explicitly, rubric hash, dataset hash) plus its results. The record is hashed and linked to the previous record's hash in an append-only JSONL ledger, so editing any past score breaks verify.

Security policy: SECURITY.md. Release notes: CHANGELOG.md. Contributing: see CONTRIBUTING.md. See DESIGN.md for what this does and does not prove.

Upgrading to 0.2: cassettes and sealed records written by 0.1.x cannot be read by 0.2, which keys entries by repeat. The demo cassette in this repo was converted in place, so the published numbers above are unchanged; your own cassettes need re-recording, and an existing ledger needs to start fresh. verify names an old record rather than calling it tampered.