Skip to main content

EvalKit

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

Full report · results in docs/results/

pip install 'evalkit-harness[anthropic,openai]'
evalkit init   # scaffold a project and a three-case sample task
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run   # estimate
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3   # needs keys in .env

Questions from PubMedQA (Jin et al., 2019, MIT licence); see DATA_CARD.md. Research benchmark, not medical advice.

Evaluate your own task across Claude and OpenAI models: accuracy, groundedness, citation validity, cost and latency side by side, with confidence intervals and two cross-family judges whose error rate is measured against held-out hand labels.

Status: v0.1.0, the first public release (on PyPI). To work on EvalKit itself, clone it and run uv sync --all-extras.

Try it in the browser: the live demo shows four models answering the same question from a study abstract, with every claim checked against the sentences it cites. Replay mode is free; live mode runs on your own API keys or the owner's passcode.

Start here: Quickstart (a first result in under five minutes) · Write your own task (JSONL or CSV) · Control cost · Calibrate the judges · Scorers and adapters · Run it in CI · Versioning

Built-in tasks: pubmedqa (yes / no / maybe research questions) and scifact (supports / refutes / not enough info for scientific claims); see DATA_CARD.md.

No telemetry: data goes only to the providers you configure (PRIVACY.md). Security and key handling: SECURITY.md. Contributions welcome: CONTRIBUTING.md.

The package is published as evalkit-harness (the name evalkit on PyPI belongs to an unrelated project). The command and the import are both evalkit.

MIT licence.

Metadata

Release files for evalkit-harness 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalkit-harness 0.2.0
File Size Uploaded Provenance Actions
evalkit_harness-0.2.0.tar.gz 198.4 kB Without provenance Details for evalkit_harness-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalkit-harness 0.2.0
File Interpreter ABI Platform Actions
evalkit_harness-0.2.0-py3-none-any.whl Python 3 none any Details for evalkit_harness-0.2.0-py3-none-any.whl

Total release size: 454.9 kB

Release files / evalkit_harness-0.2.0.tar.gz

Download URL evalkit_harness-0.2.0.tar.gz
Size 198.4 kB
Tags Source
SHA-256 checksum
How to use checksums
ab9dee7427493c2a5d8001eb3997a640060788484b8f6be0008d7b6aacd96971
BLAKE2b-256 checksum
How to use checksums
eefc677a40ce000e77e7d01089cb799a5494c63865db19dc70d8a67ee5803355
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / evalkit_harness-0.2.0-py3-none-any.whl

Download URL evalkit_harness-0.2.0-py3-none-any.whl
Size 256.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c31ab531dab69c07b5e24c94833c7f1e3b8f1f5b67aadbcd4e270fa02c3cfa92
BLAKE2b-256 checksum
How to use checksums
d291020594be4b22030a19638d3aab4e29703888d41ed2adf3545c98455622c2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for evalkit-harness 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page