evaltrust 0.2.0

An auditor for LLM evaluations — tells you whether you can trust your eval results.

EvalTrust

An auditor for LLM evaluations.
It doesn't tell you how good your model is — it tells you whether you can trust the evaluation you used to decide.

Install · Quick start · What it checks · Docs · Contributing

---

Teams spend real money evaluating models, then look at two numbers:

Model A: 84.7
Model B: 86.2   ->  ship B

That single comparison hides a dozen assumptions. Maybe the difference isn't statistically significant. Maybe the sample is too small. Maybe another judge disagrees, or the benchmark is already saturated. Most eval tools tell you what your score is. EvalTrust tells you whether you should believe it.

It works like a financial audit: bookkeeping answers "what are the numbers?"; an audit answers "can you trust them?" EvalTrust is the audit for evaluations. It runs after your existing eval tool — it doesn't replace it.

Example

$ evaltrust audit gpt4_run.json claude_run.json

EvalTrust Audit
Comparing claude-3 vs gpt-4  · 150 examples · source: deepeval+deepeval
╭─ Verdict ────────────────────────────────────────────────────────────────────╮
│ Low Confidence                                                               │
│ The evidence does not support the conclusion. Do not ship on this result     │
│ as-is — resolve the issues below first.                                      │
╰──────────────────────────────────────────────────────────────────────────────╯
 ✗ Improvement is not statistically significant   Statistical Validity
 ⚠ 95% confidence interval overlaps zero          Statistical Validity
 ⚠ Effect size is negligible                      Statistical Validity
 ✓ Benchmark has headroom                         Benchmark Health

  ✗ Improvement is not statistically significant
    Why it matters   A raw gap means nothing until you rule out chance.
    How we detected  A paired permutation test over 150 examples gave p = 0.41.
    How to fix       Do not claim a winner yet. Collect more examples first.

Two runs at 71% and 74% — a three-point "win" that is actually noise. EvalTrust catches it before it becomes a shipping decision.

Installation

pip install evaltrust

Requires Python 3.10 or newer. That's the whole setup — no API keys, no config, no account.

Install from source (for development)

git clone https://github.com/k-dickinson/evaltrust
cd evaltrust
pip install -e ".[dev]"
pytest

Quick start

1. Run your evaluation with whatever tool you already use (Promptfoo, DeepEval, or anything that can export scores to CSV/JSON) and save the output.

2. Point EvalTrust at it:

   # A file that already compares two or more models (e.g. Promptfoo):
   evaltrust audit results.json
   
   # Two single-model runs (e.g. two DeepEval runs), paired by example id:
   evaltrust audit gpt4_run.json claude_run.json
   

3. Read the verdict. Fix what it flags. Re-run.

Want to see it work before pointing it at your own data? The repo ships sample files:

evaltrust audit examples/clean_win.json        # -> High Confidence
evaltrust audit examples/borderline.json       # -> Moderate Confidence
evaltrust audit examples/deepeval_gpt4.json examples/deepeval_claude.json

Useful flags:

Flag	Effect
--json	Emit the full audit as JSON, for CI logic and experiment trackers.
--plain	Plain ASCII output — safe for Windows terminals, CI logs, and piping to a file.
--strict	Exit with a non-zero status on a Low-Confidence verdict (use it to gate CI).
--model-a, --model-b	Choose which two models to compare, or label the two files.
--alpha	Significance level (default 0.05).
--seed	Seed for the resampling (results are deterministic; change only to stress-test).

Use it from Python

The CLI is a thin wrapper over a small API, so you can audit inside a notebook, a training script, or a CI job:

import evaltrust

report = evaltrust.audit("results.json")           # path, two paths, or an EvalData
print(report.verdict.level)                         # VerdictLevel.HIGH / MODERATE / LOW

if report.verdict.level is evaltrust.VerdictLevel.LOW:
    raise SystemExit("Evaluation isn't trustworthy enough to ship on.")

report.to_dict()          # machine-readable, JSON-serializable — log it, store it, diff it

Gate CI on it

--strict returns a non-zero exit code on a Low-Confidence verdict, so an audit can block a merge the way failing tests do:

# .github/workflows/eval.yml
- name: Audit the evaluation
  run: |
    pip install evaltrust
    evaltrust audit results.json --strict --plain

What it checks

EvalTrust audits four pillars of trust and ends in one plain-language verdict — High, Moderate, or Low Confidence. There is no arbitrary aggregate score.

Pillar	The question it answers
Statistical Validity	Is the difference a real improvement, no real difference, or just too little data to tell? Significance (McNemar for pass/fail, paired permutation for continuous), equivalence testing, an interpretable effect size, and the minimum detectable effect.
Benchmark Health	Can the benchmark even separate these models, or is it saturated / flat?
Repeatability	If you reran the evaluation, would the winner stay the winner? Uses repeated-run data when the file contains it.
Judge Reliability	Would a different judge reach the same verdict? Uses multi-judge data when the file contains it.

Every finding follows the same rule — why it matters, how we detected it, and how to fix it. Checks that need extra data (repeated runs, multiple judges) don't guess when it's missing; they tell you how to generate it.

See docs/checks.md for the methods and thresholds behind each one.

Supported inputs

You never write an EvalTrust-specific format. It reads what your tool already produced and auto-detects the shape. First-class adapters today:

• Promptfoo results (several providers compared across test cases)
• DeepEval test-results export (one model per run — pass two files to compare)
• Nested JSON — {"models": [...], "examples": [{"id", "scores": {...}}]}
• Record lists — JSON like [{"id", "model", "score"}, ...]
• CSV — long (id,model,score) or wide (id,gpt,claude)

Tools without a dedicated adapter yet (LangSmith, OpenEvals, and others) work by exporting to CSV or a record list — usually a one-liner. More native adapters are a top roadmap item; contributing one is straightforward. Details and single-model pairing in docs/input-formats.md.

How it works

your eval output ──▶ auto-detect + adapter ──▶ canonical model ──▶ audit ──▶ verdict

Adapters map every format into one internal representation, so the statistics are written once and work everywhere. Every statistical method is validated in the test suite against an independent reference (scipy and statsmodels), and all resampling is seeded, so the auditor is itself reproducible. See docs/architecture.md.

Roadmap

• Now: offline CLI and Python API, four pillars, terminal / JSON / plain output, equivalence testing, adapters for Promptfoo, DeepEval, JSON, and CSV.
• Next: multi-metric eval suites (auditing several metrics at once with multiple-comparison correction), more native adapters, and an optional HTML report.
• Later: opt-in orchestration for the pillars that need to generate evidence (robustness perturbations, extra judges) and a provenance/reproducibility check.

Contributing

Contributions are welcome — new format adapters and additional checks especially. Start with CONTRIBUTING.md. All participants are expected to follow the Code of Conduct. Report security issues per the security policy.

License

EvalTrust is released under the MIT License — a permissive, OSI-approved license. Anyone, including companies and organizations, may use, modify, and distribute it, in commercial or proprietary settings, free of charge. There is no copyleft obligation and no contributor license agreement to sign.