EvalKit
Find out which model gets your task right, and whether its answers stick to the source.
EvalKit runs your own cases across Claude and OpenAI models and reports, side by side, how often each model is correct, whether every claim it makes is supported by the sentences it cites, and what that costs, with confidence intervals and two judges from different model families.
Live demo · Benchmark report · Eval index · PyPI · Quickstart · User guide · Links · Report a bug
On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.
Full report · results in docs/results/
pip install 'evalkit-harness[anthropic,openai]'
evalkit init # scaffold a project and a three-case sample task
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run # estimate
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 # needs keys in .env
Questions from PubMedQA (Jin et al., 2019, MIT licence); see DATA_CARD.md. Research benchmark, not medical advice.
What EvalKit is
An open-source Python harness for evaluating language models, and the systems built on them, on your own cases. Public leaderboards rank models on someone else's task, and a fluent answer can still say things its source never said. EvalKit measures both on your data:
- Accuracy: the share of answers with the right label (yes / no / maybe, supports / refutes, or your own labels).
- Groundedness: each answer is split into claims, and every claim is checked against the sentences the answer cites. The headline metric is correct and fully grounded.
- Citation validity, coverage, cost and latency for every model, from the providers' own usage numbers, with tokens, cache hits and cost per correct answer.
- Honest uncertainty: 95% bootstrap intervals on every rate, paired comparisons with a Holm correction, and a gap is claimed only when it survives both.
- Judges you can check: two judges from different families (Claude and GPT) score every claim, their self-preference is reported, and you calibrate them against your own hand labels.
Around the run sit the tools a team needs to keep an evaluation honest over time: release gates, run comparison, a run history with alarm bands, drift checks, held-out discipline and an agent security kit. None of them calls a model or needs a key.
Who it's for
- Teams choosing a model for a task where answers must come from a source: support, policy, research, health or legal question answering, RAG.
- Builders of agents and LLM systems who want a CI gate that fails on a real regression, not on noise, and a history that shows when a metric drifts.
- Anyone publishing eval numbers who wants intervals, cross-family judges, a calibration stamp and a cost ledger behind every figure.
Quickstart in 60 seconds
pip install 'evalkit-harness[anthropic,openai]'
evalkit init
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run
evalkit init scaffolds evalkit.yaml, a three-case sample task and .env.example. The dry
run prints what a run would cost and makes no API call:
dry run for my-task, 3 cases → runs/r-bded23433341
generate claude-sonnet-5-5 3 calls $0.0105
generate claude-haiku-5-5 3 calls $0.0005
generate gpt-6.1-sol 3 calls $0.0105
generate gpt-6-luna 3 calls $0.0005
extract_claims claude-haiku-5-5 12 calls $0.0036
judge claude-opus-5-5 54 calls $0.1242
judge gpt-6.1-sol 54 calls $0.0621
claims per item: planning default 4.5, PubMedQA answers (D-46, measured 4.46); an expected count, not an upper bound
estimated total $0.21 (no API calls made)
Add your keys to .env (copy .env.example), check the setup with evalkit doctor, then run
the same command without --dry-run. The run writes report.html and summary.json to its run
folder. No keys yet? evalkit report --example renders a sample report offline.
The package is evalkit-harness on PyPI (the name evalkit belongs to an unrelated project);
the command and the import are both evalkit. Python 3.11 to 3.13 on Linux, macOS and Windows.
Next: the user guide takes your
own task from cases to a gated, tracked evaluation.
What you get
A report per run. Each model's correct and fully grounded rate, per judge, with its 95% interval, against cost; then the models table, pairwise comparisons, judge calibration, self-preference and a failure explorer.
Gates and comparisons with no model call. evalkit gate checks release gates on a run's
summary or any project's metrics-1 file, with exact one-sided bounds on rates.
evalkit compare says, per metric and label, whether a new run is better or worse, with a 95%
interval and a Holm correction.
| Release gates | Run comparison |
|---|---|
| A metrics-1 fixture from assay-pipeline, verdict PASS. | Two fixture runs: settings that differ, each change with its interval, verdict. |
A history and a security kit. The run ledger judges each run against your own history and draws a trend page; the security kit attacks your agent and reports which defence stopped each attack.
| Ledger trend page | Security kit summary |
|---|---|
| Bands from at least five comparable runs; flags by shape and label, never colour alone. | Held-out attacks appear only as split-level aggregates. |
Every screenshot is a capture of a committed file or of an offline command on a committed fixture; none needed a key or cost anything (shot list).
What's in v0.4.0
| Capability | What it does | Entry point | Reference |
|---|---|---|---|
| Runs and judges | Your cases across Claude, OpenAI and OpenAI-compatible models; claims extracted; two judges from different families on every claim, on the batch APIs at half price; every call cached, so any stop resumes free | evalkit run |
quickstart, write a task |
| Calibration | Blind hand-label sheet, held-out Cohen's kappa per judge with a 95% interval, your own self-consistency, tied to the rubric hash and scored once; any two-way criterion with --criterion |
evalkit calibrate |
calibrate |
| Cost estimates | A dry-run quote per step and model with its claims-per-answer assumption, a hard cost cap checked before every call, free resume | evalkit run with --dry-run |
cost |
| Gates | Release gates on EvalKit summaries or any metrics-1 file: points, exact one-sided bounds, disparity between groups, warn or fail |
evalkit gate |
gates |
| Compare | Per metric and label change with a paired or unpaired 95% interval, Holm correction, a settings diff, held-out rows folded | evalkit compare |
compare |
| Ledger and drift | Hash-chained run, experiment and spend ledgers; alarm bands from your own history; a trend page; a spend cap alarm; PSI per feature and Brier, ECE and reliability-bin drift | evalkit history, evalkit experiment, evalkit spend, evalkit drift |
ledger, drift |
| Security kit | OWASP LLM Top 10 attack cases with planted canaries, k trials and pass^k, stopping-layer attribution, a model-free scripted attacker, metrics-1 output for gates | python -m evalkit.security |
security |
| Held-out discipline | A manifest of hashes, score once per rubric, an exposure scan for leaked held-out text, a tamper-evident append-only log | python -m evalkit.heldout |
heldout |
| Scorers | Label, citation validity, groundedness, coverage and per-sentence citation support; sql_match with opt-in shape equivalences (percent_scale, pivot, empty_check, period_number); value_check, which verifies "field is value" facts lines and calls prose unverifiable; your own scorers and adapters as plugins |
evalkit.scorers |
plugins |
| Contamination probe | The same PubMedQA questions reworded, original against paraphrase, to see whether accuracy rests on memorised wording | evalkit report with --paraphrase |
contamination |
| Eval index | One page with every project's headline, calibration stamp, gate status and efficiency | evalkit index |
index |
What is public API, and how releases follow semver: versioning.
How it works
flowchart LR
A[Your cases<br/>JSONL, CSV or a built-in task] --> B[Generate<br/>each model answers and cites sentences]
B --> C[Validate<br/>schema and citations]
C --> D[Extract claims<br/>one fixed extractor]
D --> E1[Judge: Claude]
D --> E2[Judge: GPT]
C --> F[Deterministic scores<br/>label, citations, cost, latency]
E1 --> G[Aggregate<br/>bootstrap CIs, thresholds]
E2 --> G
F --> G
G --> H[report.html<br/>summary.json]
- Every model call is cached by its exact inputs, so a run stopped by the cost cap, a crash or Ctrl-C resumes for free with the same command.
- Judges run on the providers' batch APIs at half price;
--live-judgesjudges immediately at full price. - A cost cap (
max_cost_usdor--max-cost) is enforced before each call; the run stops with exit code 3 rather than overspend. - Thresholds in
evalkit.yamlmake the exit code fail when a model misses the bar, so a run can gate CI.
Guides
| Guide | What it covers |
|---|---|
| User guide | Your own task end to end: cases, config, cost quote, run, report, calibration, CI gate, compare, ledger, drift |
| Quickstart | Install, sample task, first result in under five minutes |
| Write your own task | Your data as JSONL or CSV, prompts, labels, slices and thresholds |
| Control cost | Estimates, the cost cap, free resume, batch versus live judging, exit codes |
| Calibrate the judges | Hand-label claims, measure judge agreement, mark a run calibrated; any criterion |
| Gates | The metrics-1 format, gates files, exact intervals, exit codes |
| Compare two runs | Per-metric changes with intervals, pairing, Holm, held-out rows |
| Run history, experiments and spend | Alarm bands against each metric's own history, a trend page, an experiment ledger with a held-out freeze guard, a spend cap alarm |
| Drift and calibration | PSI per feature and Brier, ECE and reliability-bin changes against a frozen reference |
| Keep your held-out split honest | Manifest, score once, exposure scan, append-only log |
| Agent security evals | Attack cases, stopping layers, pass^k, scripted attacker, gates |
| Contamination probe | Original against paraphrased PubMedQA questions |
| Scorers and adapters | Add a scorer or a model adapter; sql_match equivalences; value_check facts lines; your own system |
| Run it in CI | A GitHub Actions workflow that estimates, runs, gates and uploads the report |
| The eval index | Every project side by side, built offline from snapshots |
| Versioning and upgrading per project | What is public API, semver, the 0.3 → 0.4 migration |
Built-in tasks and your own data
| Task | Data | Labels |
|---|---|---|
pubmedqa |
PubMedQA expert-labelled questions with their abstracts | yes / no / maybe |
scifact |
SciFact scientific claims with evidence abstracts | supports / refutes / not enough info |
| your own | A JSONL or CSV file: question, context sentences, expected label | your label set |
Downloads are verified against pinned checksums; licences and sampling are in DATA_CARD.md.
Used by
Seven projects build on EvalKit, each pinned in downstream.tsv. A canary runs each project's own EvalKit tests against every release candidate, and the eval index shows them side by side.
| Project | What it is | What it uses from EvalKit |
|---|---|---|
| agentic-analytics | A ClickHouse MCP agent (Claude, GPT) against Snowflake Cortex Analyst on a 100-question analytics benchmark | sql_match and its shape equivalences, clustered intervals, cost estimates |
| ai-visibility | Which websites Claude and GPT cite for UK health questions, and how fresh those pages are | Two-judge support checks of whether cited pages state the claim |
| assay-pipeline | EGFR bioactivity data read by an AI agent through a SQL guard and a hash-chained audit log, attacked with OWASP LLM Top 10 cases | sql_match, attack outcomes and gates; its attack suite became the security kit |
| course-assistant | A course assistant that answers only from approved session materials and cites every slide | System adapter, task scorers, slices, calibration of its integrity criterion |
| fraud-review-bedrock | A fraud-review agent on Amazon Bedrock that explains each decision against a written policy (simulated data) | Faithfulness, cross-family judges, paired bootstrap with Holm, value_check, PSI drift |
| gemma-grounded-health | A small open Gemma model fine-tuned to answer health questions only from a given abstract and cite it | The PubMedQA task, answer parsing, judge identity, the summary format |
| respectful-offers | A subscription offer policy with a contextual bandit and two respect rules, evaluated offline | A metrics-1 export gated with evalkit gate, drift checks at warn severity |
How each project moves to 0.4: the per-project guide.
The live demo
evalkit-demo.streamlit.app shows the idea on one question at a time.
Your keys go only into that request's API clients: never saved, never logged. Live requests are
rate-limited per visitor and capped per request; only aggregate counts are logged, never the
question or the abstract. The demo's source is in
demo/.
Status: what is live, and what is not
- Released: v0.4.0 on PyPI (2026-11-18), by trusted publishing from a tag, with CI green and a post-release install check on Linux, macOS and Windows.
- Live: the benchmark report on GitHub Pages, the demo (replay open, live behind a passcode), and the cross-project eval index, which is being published to the same site.
- Provisional: the benchmark's judges are not yet calibrated against hand labels, so
its report carries the "judge not calibrated for this task" stamp and its numbers are
provisional. The hand labels are the open item; until they are done, no headline is
published as calibrated (
evalkit report --publishrefuses). - Downstream: the v0.4.0 release-candidate canary passed for three of the seven projects; the other four fail only known-drift checks (version-pin tests, a byte-for-byte summary fixture, a config-spec test), each explained in the canary record and fixed in that project's own build when it moves to 0.4.
- Spend: the 200-question benchmark cost $23.87 in API calls and the whole project $31.56, every dollar in the spend ledger.
Results
The first benchmark (200 PubMedQA questions, four models, two judges) is in the
full report, with the raw summary in
docs/results/. How it was
built and what was decided along the way: the
case study.
Links
| Repository | https://github.com/biplob819/evalkit |
| Package | https://pypi.org/project/evalkit-harness/ |
| Benchmark report | https://biplob819.github.io/evalkit/ |
| Live demo | https://evalkit-demo.streamlit.app/ |
| Cross-project eval index | https://biplob819.github.io/evalkit/index/ |
| User guide | docs/guide.md |
| Docs | docs/ |
| Case study | docs/case-study.md |
| Decision log | docs/decisions.md |
| Lessons | docs/lessons.md |
| Changelog | changelog.md |
| Product requirements and tech spec | EvalKit PRD.md, EvalKit Tech Spec.md |
| Data card | DATA_CARD.md |
| Walkthrough shot list | docs/walkthrough.md |
| Contributing | CONTRIBUTING.md |
| Code of conduct | CODE_OF_CONDUCT.md |
| Security | SECURITY.md |
| Privacy | PRIVACY.md |
| Licence | LICENSE (MIT) |
| Issues | https://github.com/biplob819/evalkit/issues/new/choose |
Build status: uv run python scripts/status.py (the project was built node by node, each
verified against written criteria and an independent reviewer).
Contributing, security and privacy
- No telemetry. Data goes only to the model providers you configure: PRIVACY.md.
- Security: key handling and how to report a vulnerability privately: SECURITY.md.
- Contributing: set-up, checks and conventions in CONTRIBUTING.md; please follow the code of conduct. Bug reports and ideas: open an issue.
- To work on EvalKit itself: clone the repository and run
uv sync --all-extras.
Credits and licence
Questions from PubMedQA (Jin et al., 2019, MIT licence) and claims from SciFact (Wadden et al., 2020); see DATA_CARD.md. EvalKit is a research tool, not medical advice.
MIT licence. See LICENSE.
Metadata
Release files for evalkit-harness 0.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evalkit_harness-0.4.1.tar.gz | 509.2 kB | Without provenance | Details for evalkit_harness-0.4.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evalkit_harness-0.4.1-py3-none-any.whl | Python 3 | none | any | Details for evalkit_harness-0.4.1-py3-none-any.whl |
Total release size: 1.1 MB
Release files / evalkit_harness-0.4.1.tar.gz
| Download URL | evalkit_harness-0.4.1.tar.gz |
|---|---|
| Size | 509.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
26ea7a9d1ade05564c7f1ad460c60fae051bb9d7b74b5c1e5e050e3ffd9e9389
|
|
BLAKE2b-256 checksum How to use checksums |
5f845c78b75af8184bb1eefa15fa00defe8e1388ae7968d4a6b8470c8596566b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / evalkit_harness-0.4.1-py3-none-any.whl
| Download URL | evalkit_harness-0.4.1-py3-none-any.whl |
|---|---|
| Size | 572.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b260c5cf05c91f567bfa90e190879061e2467afcf2c61d63eb80dd5a66f95f7e
|
|
BLAKE2b-256 checksum How to use checksums |
c3931d8109d2609275ccfec1f35b554d1b1e87796a09fa7eb822472923fb850b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for evalkit-harness 0.4.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.