Skip to main content

evalproof

ci licence Open in Streamlit

Documentation: basantbhattarai.com.np/evalproof

Treat LLM evaluation as measurement. Calibrate the judge, correct its pass rate, and gate CI on regressions that are real rather than noise.

Try it in the browser: evalproof.streamlit.app. Check whether a drop between two runs is real, or replay the measured run and read the twelve answers that changed. It runs this repository's code; after a quiet spell it takes about half a minute to wake.

An LLM judge is a classifier with error rates. If it passes bad answers at rate FPR and good answers at rate TPR, its pass rate converges to FPR + (TPR − FPR) · p, not to the true rate p. That is classify-and-count, and it is biased the way any classifier's count is — the effect is worked through with a seeded simulation in The classifier was 95% accurate. The trend it counted was a third too flat (lab).

evalproof:

  1. measures the judge against a small gold-labelled sample — confusion matrix, TPR, FPR, Cohen's κ;
  2. corrects its pass rate with adjusted classify-and-count and prediction-powered inference, each with a confidence interval;
  3. compares runs on paired cases — exact McNemar for pass/fail, paired bootstrap for scores, and a paired permutation test when each case runs several times;
  4. gates CI only when a drop is both larger than your tolerance and statistically significant, plus latency and cost budgets, with distinct exit codes;
  5. checks the run files it gates — verify proves, with no model call, that a committed run was made with this suite and covers the dataset as it is now;
  6. tests agents — asserts on the tools an agent called and what it said, across several turns, several builds of the same system and repeated trials, with adapters for ADK, LangChain/LangGraph, the Claude Agent SDK and OpenAI Agents (Testing agents).

It is local-first: Ollama is the default, every major hosted provider is one target: line away (Providers), a logged-in Claude Code or Codex can be the judge with no API key, and every model call is cached by content so re-runs are free and reproducible. Results from promptfoo, Inspect, DeepEval, lm-eval, garak, Google ADK and Claude Code plugin evals can be imported and gated without re-running them. The test cases you already have in promptfoo, DeepEval, Ragas, LangSmith, OpenAI Evals, Inspect, Braintrust, Langfuse, Phoenix or a Google ADK eval set can be turned into a dataset and a suite with evalproof import-dataset auto FILE --out cases.jsonl --suite-out suite.yaml, or when you set up with evalproof init --from FILE (Import test cases).

Install

evalproof is a command-line tool and a Python library, published on PyPI. It needs Python 3.11 to 3.14 and has four runtime dependencies (httpx, numpy, pydantic, PyYAML); no vendor SDK, no optional extras.

Run it without installing (needs uv):

uvx evalproof --version
uvx evalproof@1.2.0 init          # pin a version

Install the CLI on your PATH, isolated from your projects:

uv tool install evalproof         # or: pipx install evalproof
evalproof --version               # evalproof 1.2.0

Add it to a project, to call it from Python or pin it with the project's other dev tools:

uv add --dev evalproof            # or: pip install evalproof
uv run evalproof --version

Upgrade with uv tool upgrade evalproof, pipx upgrade evalproof or pip install -U evalproof. Run files from older versions keep working; see Upgrading for what to change and the changelog for everything that changed.

In a container: docker run --rm -v "$PWD:/work" ghcr.io/technobasant/evalproof:latest --version (Python 3.13, linux/amd64 and linux/arm64; no coding-agent CLIs in it, see Docker).

In GitHub Actions, no install step is needed: the composite action technobasant/evalproof@v1.2.0 runs the evalproof of the tag you pin, and pr-comment: "true" posts the report as a pull request comment. In GitLab CI, evalproof init --platform gitlab writes a file you include from .gitlab-ci.yml (In CI).

What it needs to run

Which credential a run needs depends on what the system and the judge are. Keyless mode needs no secrets: CI compares run files you committed and makes no model call. Fork pull requests never receive secrets, so no workflow runs a model for them.

Target Credential variable Local use CI use Where allowed
No model: a Python function or an HTTP endpoint with no LLM behind it, or committed run files none nothing nothing anywhere, including fork pull requests
Local models (Ollama, vLLM, LM Studio, llama.cpp) none the server running, with the model pulled a server on the runner, started in a workflow step anywhere
Hosted APIs (Anthropic, OpenAI and the rest of Providers) the provider's key, such as ANTHROPIC_API_KEY or OPENAI_API_KEY the variable in your environment the key as a repository secret your own repository; fork pull requests never receive it
Claude Code CLI ANTHROPIC_API_KEY (api-key mode) or CLAUDE_CODE_OAUTH_TOKEN your Claude Code login, no variable ANTHROPIC_API_KEY as a secret with EVALPROOF_CLI_AUTH=api-key, or CLAUDE_CODE_OAUTH_TOKEN as a secret (made by claude setup-token) the token: your own repository and your own CI only; use an API key for a secret shared across a team or an organisation
Codex CLI OPENAI_API_KEY your Codex login, no variable OPENAI_API_KEY as a secret with EVALPROOF_CLI_AUTH=api-key a personal Codex login only on trusted private infrastructure, never on public repos or for open source work

evalproof init --ci api-key writes the workflow with the secret for the judge it chose, and prints this requirement for that judge when it finishes. For Claude Code, evalproof init --ci api-key --cli-auth oauth-token passes CLAUDE_CODE_OAUTH_TOKEN instead of the API key and warns that it is for your own repository and CI only; Codex has no token option. The full page adds what evalproof doctor reports.

Use cases

Each is a suite file and one or two commands (use cases): gate a prompt or model change in a pull request; choose between two models or providers; find out whether your LLM judge can be trusted; use your Claude Code or Codex subscription as the judge; gate results you already have from another eval tool; check a retrieval or RAG change; hold latency and cost while you swap a model; gate in CI with no model calls and no secrets; test a non-LLM system the same way.

Five-minute quickstart

In your own repository, one command writes a starter suite, a CI workflow and an agent skill, and a second runs it:

uvx evalproof init
uvx evalproof run evals/suite.yaml --out evals/runs/latest.json

init picks the judge from what is on your PATH: a logged-in claude, then codex, then Ollama, then a hosted Anthropic model with ANTHROPIC_API_KEY. It writes evals/ (the suite, three sample cases, a README), .github/workflows/evalproof.yml (--ci keyless, the default, makes no model calls in CI; --ci api-key runs the suite there; --ci none writes no workflow) and a skill for coding agents at .agents/skills/evalproof/SKILL.md and .claude/skills/evalproof/SKILL.md (--no-skill skips it). It also adds an evalproof section to your AGENTS.md, between two markers, so any coding agent that reads that file knows which command to run for a task, how to read the exit codes and what never to weaken (--no-agents-md skips it; --agent-files-only writes only the section and the skill). Your own text in that file is never changed. It prints one line per file, never overwrites one without --force, and makes no model call. Replace the sample cases with your own: the gate needs 20 before it will decide.

The offline example, the retrieval example and the LLM example are on the quickstart page.

Documentation

The site at basantbhattarai.com.np/evalproof has:

Changelog · Contributing · Security policy · Code of conduct · How to cite. Releases are published to PyPI from GitHub Actions with trusted publishing, and each file carries a signed provenance attestation.

Licence

Apache-2.0. See LICENSE and NOTICE.

Metadata

Release files for evalproof 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalproof 1.2.0
File Size Uploaded Provenance Actions
evalproof-1.2.0.tar.gz 560.8 kB With provenance Details for evalproof-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalproof 1.2.0
File Interpreter ABI Platform Actions
evalproof-1.2.0-py3-none-any.whl Python 3 none any Details for evalproof-1.2.0-py3-none-any.whl

Total release size: 861.5 kB

Release files / evalproof-1.2.0.tar.gz

Download URL evalproof-1.2.0.tar.gz
Size 560.8 kB
Tags Source
SHA-256 checksum
How to use checksums
d6c8543101a3e66b94ecf4f08945feba8a820b25945552b105e23a06724e9285
BLAKE2b-256 checksum
How to use checksums
daf0f2bb1ee1ece51d88dc2453b3a20341a6a2bb6eea7ca40a48546f598bef9c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / evalproof-1.2.0-py3-none-any.whl

Download URL evalproof-1.2.0-py3-none-any.whl
Size 300.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
816aec4c35ac3b34c2f80895e3ee7276b72cd3fb3d4d55801a8140d67eec1485
BLAKE2b-256 checksum
How to use checksums
cfd582d0a125f2bef4cf3ac4ffe99bece841f1a206bedcb164668e9f31c51f35
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Security for evalproof 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for evalproof-1.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page