evalwarden

A linter for evals, not another eval framework.

Benchmark scores get cited as proof of capability: which model to deploy, which paper to accept, which agent to buy. Almost nobody audits the benchmarks themselves. A solver can read the task ID from the environment, look up the gold answer, and report 100%. A grader can be writable by the agent it grades. A model judge can be uncalibrated, biased, and self-contradictory — while two-thirds of the dataset is dead weight every model already answers. The score looks fine. The score is meaningless.

evalwarden audits the measurement system around a score: the dataset, the evidence boundary, the grader, the run records, and the cost. It never runs your evals and never changes your harness. It reads your eval artifacts and tells you whether the score can be trusted, with file-level evidence for every finding.

Quickstart

pip install evalwarden
evalwarden demo

That audits a deliberately broken benchmark and writes evalwarden-demo-report.html. Open it in a browser. No model key required.

The flagship demo: caught red-handed

A tiny synthetic coding benchmark reports 3/3 PASS. The solver earned none of it: it reads TASK_ID from the environment, looks up the answer in gold_map.json, and submits the gold patch. The auditor flags the exact leak channels, each with file-level evidence:

• ENV-001 — TASK_ID, RUN_ID, AGENT_TOKEN visible to the agent
• ENV-001 — gold_map.json mounted where the agent can read it
• GRAD-001 — the verifier is writable by the agent

Result: 0/100 BLOCKED. Then the hardened fixture shows the fix: opaque IDs, no gold mounted, read-only verifier, a genuine solver. Result: 100/100 PASS. Same benchmark, same auditor, before and after.

evalwarden demo --fixture leaky     # the cheat: 0/100 BLOCKED
evalwarden demo --fixture hardened  # the fix: 100/100 PASS

The check catalog

Deterministic, high-precision checks. A linter that cries contamination on a clean eval is worse than no auditor, so every finding carries a confidence label and clean evals produce zero findings.

ENV — the evidence boundary

GRAD — the grader

COST — the run records

JUDGE — the model judge

DATA — the dataset itself

TRAJ — the agent's trajectory

evalwarden explain COST-004 prints any check's threat model, evidence, and fix.

How evalwarden differs

Eval frameworks run evals. evalwarden audits them. Inspect AI, Promptfoo, and lm-eval-harness execute tasks and compute scores. evalwarden never executes anything: read-only adapters translate harness artifacts into a framework-neutral integrity model, and checks only ever see that model. That boundary is what keeps this a linter instead of another framework.

Leaderboards rank models. evalwarden grades the tests. A leaderboard tells you who won on a benchmark. evalwarden tells you whether the benchmark was worth winning on — whether the dataset still discriminates, the judge is calibrated, and the evidence boundary held.

One-off audit notebooks don't run in CI. These checks do. A notebook audit is a snapshot that rots. evalwarden's checks are deterministic rules with policy exit codes (0 passes, 1 findings cross --fail-on), so the same audit runs on every commit.

Precision over recall, always. A false contamination accusation is worse than a missed issue. Checks stay silent when the evidence is thin, every finding carries a confidence label, and every card says "Diagnostic, not a certification" — the report says "no blocking findings observed under this policy," never "certified safe."

Proof on real benchmarks

The linter runs against real public benchmarks, translated mechanically from their published definitions: pinned sources, no invented traces, provenance committed alongside. Cards live in examples/report-cards/real/:

The reading guide walks through the cards, the translation, and the scope limits.

Evidence registry

Report cards live on as versioned, independently regenerable evidence in the Evalwarden Evidence Registry: each entry is an evidence pack binding an executable audit to pinned public inputs and a one-command regeneration, so a stranger can verify the card.

How it works

eval artifact/ ──▶ adapter (inspect | promptfoo) ──▶ integrity model ──▶ checks ──▶ report
     read-only, offline                          data · boundary · grader · runs

Two adapters ship: inspect for Inspect-style eval artifact directories, and promptfoo for Promptfoo's promptfooconfig.yaml plus the JSON export from promptfoo eval --output results.json. Both are strictly read-only and offline; variable names are kept for analysis while secret values never enter the normalized model. A new check is one module plus one registration line; a new reporter is one module plus one import.

CLI

evalwarden audit <eval-artifact> [--adapter auto|inspect|promptfoo] [--output report.html]
                              [--json findings.json] [--fail-on high]
                              [--price-in 3.0] [--price-out 15.0]
                              [--budget-per-task USD]
evalwarden demo [--fixture leaky|hardened|judge_bad|judge_clean|cost_wasteful|cost_clean|promptfoo_bad|promptfoo_clean]
             [--output evalwarden-demo-report.html] [--budget-per-task USD]
evalwarden report-card <eval-artifact> [--output card.html]
evalwarden report-cards <eval...> [--fixtures a,b] [--output-dir cards/]
evalwarden explain <CHECK-ID>

Exit codes: 0 policy passes, 1 findings cross --fail-on, 2 the audit could not complete. The same policy runs locally and as a CI gate.

Reports

Self-contained HTML (inline CSS, no JavaScript, no remote assets), terminal output, JSON findings, and report cards. Secret values are never stored — only variable names enter the model. Integrity scores are diagnostic, not a certification.

Non-goals

Running or scheduling evaluations, replacing task/solver/scorer APIs, trace observability, generic red-teaming, public leaderboards, declaring any benchmark contamination-free.

Development

pip install -e ".[dev]"
pytest

The test suite is the product's credibility: every rule has positive, negative, and precision fixtures (clean evals must not be flagged), the adapter has a read-only contract test, and the fixtures run end to end.