Skip to main content

EvalKit

PyPI Python CI Licence: MIT Live demo

Find out which model gets your task right, and whether its answers stick to the source.

EvalKit runs your own cases across Claude and OpenAI models and reports, side by side, how often each model is correct, whether every claim it makes is supported by the sentences it cites, and what that costs, with confidence intervals and two judges from different model families.

Live demo · Benchmark report · Eval index · PyPI · Quickstart · User guide · Links · Report a bug

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

Full report · results in docs/results/

pip install 'evalkit-harness[anthropic,openai]'
evalkit init   # scaffold a project and a three-case sample task
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run   # estimate
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3   # needs keys in .env

Questions from PubMedQA (Jin et al., 2019, MIT licence); see DATA_CARD.md. Research benchmark, not medical advice.

Walkthrough: the headline, the report and its failure explorer, a dry-run cost quote, release gates, a run comparison, the ledger trend, the security kit, the downstream canary and the decision log

What EvalKit is

An open-source Python harness for evaluating language models, and the systems built on them, on your own cases. Public leaderboards rank models on someone else's task, and a fluent answer can still say things its source never said. EvalKit measures both on your data:

  • Accuracy: the share of answers with the right label (yes / no / maybe, supports / refutes, or your own labels).
  • Groundedness: each answer is split into claims, and every claim is checked against the sentences the answer cites. The headline metric is correct and fully grounded.
  • Citation validity, coverage, cost and latency for every model, from the providers' own usage numbers, with tokens, cache hits and cost per correct answer.
  • Honest uncertainty: 95% bootstrap intervals on every rate, paired comparisons with a Holm correction, and a gap is claimed only when it survives both.
  • Judges you can check: two judges from different families (Claude and GPT) score every claim, their self-preference is reported, and you calibrate them against your own hand labels.

Around the run sit the tools a team needs to keep an evaluation honest over time: release gates, run comparison, a run history with alarm bands, drift checks, held-out discipline and an agent security kit. None of them calls a model or needs a key.

Who it's for

  • Teams choosing a model for a task where answers must come from a source: support, policy, research, health or legal question answering, RAG.
  • Builders of agents and LLM systems who want a CI gate that fails on a real regression, not on noise, and a history that shows when a metric drifts.
  • Anyone publishing eval numbers who wants intervals, cross-family judges, a calibration stamp and a cost ledger behind every figure.

Quickstart in 60 seconds

pip install 'evalkit-harness[anthropic,openai]'
evalkit init
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run

evalkit init scaffolds evalkit.yaml, a three-case sample task and .env.example. The dry run prints what a run would cost and makes no API call:

dry run for my-task, 3 cases → runs/r-bded23433341
generate        claude-sonnet-5-5           3 calls  $0.0105
generate        claude-haiku-5-5            3 calls  $0.0005
generate        gpt-6.1-sol                 3 calls  $0.0105
generate        gpt-6-luna                  3 calls  $0.0005
extract_claims  claude-haiku-5-5           12 calls  $0.0036
judge           claude-opus-5-5            54 calls  $0.1242
judge           gpt-6.1-sol                54 calls  $0.0621
claims per item: planning default 4.5, PubMedQA answers (D-46, measured 4.46); an expected count, not an upper bound
estimated total $0.21 (no API calls made)

Add your keys to .env (copy .env.example), check the setup with evalkit doctor, then run the same command without --dry-run. The run writes report.html and summary.json to its run folder. No keys yet? evalkit report --example renders a sample report offline.

The package is evalkit-harness on PyPI (the name evalkit belongs to an unrelated project); the command and the import are both evalkit. Python 3.11 to 3.13 on Linux, macOS and Windows. Next: the user guide takes your own task from cases to a gated, tracked evaluation.

What you get

A report per run. Each model's correct and fully grounded rate, per judge, with its 95% interval, against cost; then the models table, pairwise comparisons, judge calibration, self-preference and a failure explorer.

The headline chart The failure explorer
Headline chart: correct and fully grounded rate per model and judge, with 95% intervals, against cost per 100 cases Failure explorer: one failed answer with its claims, each judge's verdict and the cited sentences
Four models on 200 PubMedQA questions, two judges, cost on a log scale. Every answer that failed for a judge, with its claims, each judge's verdict and the cited sentences.

Gates and comparisons with no model call. evalkit gate checks release gates on a run's summary or any project's metrics-1 file, with exact one-sided bounds on rates. evalkit compare says, per metric and label, whether a new run is better or worse, with a 95% interval and a Holm correction.

Release gates Run comparison
evalkit gate: one row per gate and matched metric with bound, threshold, observed value, severity and result, then the verdict evalkit compare: per metric and label, old and new values, the change with its 95% interval, pairing and flag, then the verdict
A metrics-1 fixture from assay-pipeline, verdict PASS. Two fixture runs: settings that differ, each change with its interval, verdict.

A history and a security kit. The run ledger judges each run against your own history and draws a trend page; the security kit attacks your agent and reports which defence stopped each attack.

Ledger trend page Security kit summary
Trend page: a metric over seven runs with median line, warn and alarm bands, settings markers and an alarm on the last run Security kit summary: pass^k, trial success rate and blocked rate per split, stopping layers and public categories
Bands from at least five comparable runs; flags by shape and label, never colour alone. Held-out attacks appear only as split-level aggregates.

Every screenshot is a capture of a committed file or of an offline command on a committed fixture; none needed a key or cost anything (shot list).

What's in v0.4.0

Capability What it does Entry point Reference
Runs and judges Your cases across Claude, OpenAI and OpenAI-compatible models; claims extracted; two judges from different families on every claim, on the batch APIs at half price; every call cached, so any stop resumes free evalkit run quickstart, write a task
Calibration Blind hand-label sheet, held-out Cohen's kappa per judge with a 95% interval, your own self-consistency, tied to the rubric hash and scored once; any two-way criterion with --criterion evalkit calibrate calibrate
Cost estimates A dry-run quote per step and model with its claims-per-answer assumption, a hard cost cap checked before every call, free resume evalkit run with --dry-run cost
Gates Release gates on EvalKit summaries or any metrics-1 file: points, exact one-sided bounds, disparity between groups, warn or fail evalkit gate gates
Compare Per metric and label change with a paired or unpaired 95% interval, Holm correction, a settings diff, held-out rows folded evalkit compare compare
Ledger and drift Hash-chained run, experiment and spend ledgers; alarm bands from your own history; a trend page; a spend cap alarm; PSI per feature and Brier, ECE and reliability-bin drift evalkit history, evalkit experiment, evalkit spend, evalkit drift ledger, drift
Security kit OWASP LLM Top 10 attack cases with planted canaries, k trials and pass^k, stopping-layer attribution, a model-free scripted attacker, metrics-1 output for gates python -m evalkit.security security
Held-out discipline A manifest of hashes, score once per rubric, an exposure scan for leaked held-out text, a tamper-evident append-only log python -m evalkit.heldout heldout
Scorers Label, citation validity, groundedness, coverage and per-sentence citation support; sql_match with opt-in shape equivalences (percent_scale, pivot, empty_check, period_number); value_check, which verifies "field is value" facts lines and calls prose unverifiable; your own scorers and adapters as plugins evalkit.scorers plugins
Contamination probe The same PubMedQA questions reworded, original against paraphrase, to see whether accuracy rests on memorised wording evalkit report with --paraphrase contamination
Eval index One page with every project's headline, calibration stamp, gate status and efficiency evalkit index index

What is public API, and how releases follow semver: versioning.

How it works

flowchart LR
  A[Your cases<br/>JSONL, CSV or a built-in task] --> B[Generate<br/>each model answers and cites sentences]
  B --> C[Validate<br/>schema and citations]
  C --> D[Extract claims<br/>one fixed extractor]
  D --> E1[Judge: Claude]
  D --> E2[Judge: GPT]
  C --> F[Deterministic scores<br/>label, citations, cost, latency]
  E1 --> G[Aggregate<br/>bootstrap CIs, thresholds]
  E2 --> G
  F --> G
  G --> H[report.html<br/>summary.json]
  • Every model call is cached by its exact inputs, so a run stopped by the cost cap, a crash or Ctrl-C resumes for free with the same command.
  • Judges run on the providers' batch APIs at half price; --live-judges judges immediately at full price.
  • A cost cap (max_cost_usd or --max-cost) is enforced before each call; the run stops with exit code 3 rather than overspend.
  • Thresholds in evalkit.yaml make the exit code fail when a model misses the bar, so a run can gate CI.

Guides

Guide What it covers
User guide Your own task end to end: cases, config, cost quote, run, report, calibration, CI gate, compare, ledger, drift
Quickstart Install, sample task, first result in under five minutes
Write your own task Your data as JSONL or CSV, prompts, labels, slices and thresholds
Control cost Estimates, the cost cap, free resume, batch versus live judging, exit codes
Calibrate the judges Hand-label claims, measure judge agreement, mark a run calibrated; any criterion
Gates The metrics-1 format, gates files, exact intervals, exit codes
Compare two runs Per-metric changes with intervals, pairing, Holm, held-out rows
Run history, experiments and spend Alarm bands against each metric's own history, a trend page, an experiment ledger with a held-out freeze guard, a spend cap alarm
Drift and calibration PSI per feature and Brier, ECE and reliability-bin changes against a frozen reference
Keep your held-out split honest Manifest, score once, exposure scan, append-only log
Agent security evals Attack cases, stopping layers, pass^k, scripted attacker, gates
Contamination probe Original against paraphrased PubMedQA questions
Scorers and adapters Add a scorer or a model adapter; sql_match equivalences; value_check facts lines; your own system
Run it in CI A GitHub Actions workflow that estimates, runs, gates and uploads the report
The eval index Every project side by side, built offline from snapshots
Versioning and upgrading per project What is public API, semver, the 0.3 → 0.4 migration

Built-in tasks and your own data

Task Data Labels
pubmedqa PubMedQA expert-labelled questions with their abstracts yes / no / maybe
scifact SciFact scientific claims with evidence abstracts supports / refutes / not enough info
your own A JSONL or CSV file: question, context sentences, expected label your label set

Downloads are verified against pinned checksums; licences and sampling are in DATA_CARD.md.

Used by

Seven projects build on EvalKit, each pinned in downstream.tsv. A canary runs each project's own EvalKit tests against every release candidate, and the eval index shows them side by side.

Project What it is What it uses from EvalKit
agentic-analytics A ClickHouse MCP agent (Claude, GPT) against Snowflake Cortex Analyst on a 100-question analytics benchmark sql_match and its shape equivalences, clustered intervals, cost estimates
ai-visibility Which websites Claude and GPT cite for UK health questions, and how fresh those pages are Two-judge support checks of whether cited pages state the claim
assay-pipeline EGFR bioactivity data read by an AI agent through a SQL guard and a hash-chained audit log, attacked with OWASP LLM Top 10 cases sql_match, attack outcomes and gates; its attack suite became the security kit
course-assistant A course assistant that answers only from approved session materials and cites every slide System adapter, task scorers, slices, calibration of its integrity criterion
fraud-review-bedrock A fraud-review agent on Amazon Bedrock that explains each decision against a written policy (simulated data) Faithfulness, cross-family judges, paired bootstrap with Holm, value_check, PSI drift
gemma-grounded-health A small open Gemma model fine-tuned to answer health questions only from a given abstract and cite it The PubMedQA task, answer parsing, judge identity, the summary format
respectful-offers A subscription offer policy with a contextual bandit and two respect rules, evaluated offline A metrics-1 export gated with evalkit gate, drift checks at warn severity

How each project moves to 0.4: the per-project guide.

The live demo

evalkit-demo.streamlit.app shows the idea on one question at a time.

Replay mode Live mode
Demo replay: four models answer the same question with their cited sentences checked Demo live mode: passcode, optional own API keys, question and abstract
Open to everyone: prepared answers to sample PubMedQA questions, no model calls. Needs a passcode from the owner, then runs on the owner's keys (within a daily budget) or on your own keys.

Your keys go only into that request's API clients: never saved, never logged. Live requests are rate-limited per visitor and capped per request; only aggregate counts are logged, never the question or the abstract. The demo's source is in demo/.

Status: what is live, and what is not

  • Released: v0.4.0 on PyPI (2026-11-18), by trusted publishing from a tag, with CI green and a post-release install check on Linux, macOS and Windows.
  • Live: the benchmark report on GitHub Pages, the demo (replay open, live behind a passcode), and the cross-project eval index, which is being published to the same site.
  • Provisional: the benchmark's judges are not yet calibrated against hand labels, so its report carries the "judge not calibrated for this task" stamp and its numbers are provisional. The hand labels are the open item; until they are done, no headline is published as calibrated (evalkit report --publish refuses).
  • Downstream: the v0.4.0 release-candidate canary passed for three of the seven projects; the other four fail only known-drift checks (version-pin tests, a byte-for-byte summary fixture, a config-spec test), each explained in the canary record and fixed in that project's own build when it moves to 0.4.
  • Spend: the 200-question benchmark cost $23.87 in API calls and the whole project $31.56, every dollar in the spend ledger.

Results

The first benchmark (200 PubMedQA questions, four models, two judges) is in the full report, with the raw summary in docs/results/. How it was built and what was decided along the way: the case study.

Repository https://github.com/biplob819/evalkit
Package https://pypi.org/project/evalkit-harness/
Benchmark report https://biplob819.github.io/evalkit/
Live demo https://evalkit-demo.streamlit.app/
Cross-project eval index https://biplob819.github.io/evalkit/index/
User guide docs/guide.md
Docs docs/
Case study docs/case-study.md
Decision log docs/decisions.md
Lessons docs/lessons.md
Changelog changelog.md
Product requirements and tech spec EvalKit PRD.md, EvalKit Tech Spec.md
Data card DATA_CARD.md
Walkthrough shot list docs/walkthrough.md
Contributing CONTRIBUTING.md
Code of conduct CODE_OF_CONDUCT.md
Security SECURITY.md
Privacy PRIVACY.md
Licence LICENSE (MIT)
Issues https://github.com/biplob819/evalkit/issues/new/choose

Build status: uv run python scripts/status.py (the project was built node by node, each verified against written criteria and an independent reviewer).

Contributing, security and privacy

  • No telemetry. Data goes only to the model providers you configure: PRIVACY.md.
  • Security: key handling and how to report a vulnerability privately: SECURITY.md.
  • Contributing: set-up, checks and conventions in CONTRIBUTING.md; please follow the code of conduct. Bug reports and ideas: open an issue.
  • To work on EvalKit itself: clone the repository and run uv sync --all-extras.

Credits and licence

Questions from PubMedQA (Jin et al., 2019, MIT licence) and claims from SciFact (Wadden et al., 2020); see DATA_CARD.md. EvalKit is a research tool, not medical advice.

MIT licence. See LICENSE.

Metadata

Release files for evalkit-harness 0.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalkit-harness 0.4.1
File Size Uploaded Provenance Actions
evalkit_harness-0.4.1.tar.gz 509.2 kB Without provenance Details for evalkit_harness-0.4.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalkit-harness 0.4.1
File Interpreter ABI Platform Actions
evalkit_harness-0.4.1-py3-none-any.whl Python 3 none any Details for evalkit_harness-0.4.1-py3-none-any.whl

Total release size: 1.1 MB

Release files / evalkit_harness-0.4.1.tar.gz

Download URL evalkit_harness-0.4.1.tar.gz
Size 509.2 kB
Tags Source
SHA-256 checksum
How to use checksums
26ea7a9d1ade05564c7f1ad460c60fae051bb9d7b74b5c1e5e050e3ffd9e9389
BLAKE2b-256 checksum
How to use checksums
5f845c78b75af8184bb1eefa15fa00defe8e1388ae7968d4a6b8470c8596566b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / evalkit_harness-0.4.1-py3-none-any.whl

Download URL evalkit_harness-0.4.1-py3-none-any.whl
Size 572.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b260c5cf05c91f567bfa90e190879061e2467afcf2c61d63eb80dd5a66f95f7e
BLAKE2b-256 checksum
How to use checksums
c3931d8109d2609275ccfec1f35b554d1b1e87796a09fa7eb822472923fb850b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for evalkit-harness 0.4.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page