Skip to main content

EvalKit

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.

Full report · results in docs/results/

pip install 'evalkit-harness[anthropic,openai]'
evalkit init   # scaffold a project and a three-case sample task
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run   # estimate
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3   # needs keys in .env

Questions from PubMedQA (Jin et al., 2019, MIT licence); see DATA_CARD.md. Research benchmark, not medical advice.

Evaluate your own task across Claude and OpenAI models: accuracy, groundedness, citation validity, cost and latency side by side, with confidence intervals and two cross-family judges whose error rate is measured against held-out hand labels.

Status: v0.1.0, the first public release. To work on EvalKit itself, clone it and run uv sync --all-extras.

Start here: Quickstart (a first result in under five minutes) · Write your own task (JSONL or CSV) · Control cost · Calibrate the judges · Scorers and adapters · Run it in CI · Versioning

Built-in tasks: pubmedqa (yes / no / maybe research questions) and scifact (supports / refutes / not enough info for scientific claims); see DATA_CARD.md.

No telemetry: data goes only to the providers you configure (PRIVACY.md). Security and key handling: SECURITY.md. Contributions welcome: CONTRIBUTING.md.

The package is published as evalkit-harness (the name evalkit on PyPI belongs to an unrelated project). The command and the import are both evalkit.

MIT licence.

Metadata

Release files for evalkit-harness 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalkit-harness 0.1.0
File Size Uploaded Provenance Actions
evalkit_harness-0.1.0.tar.gz 190.2 kB Without provenance Details for evalkit_harness-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalkit-harness 0.1.0
File Interpreter ABI Platform Actions
evalkit_harness-0.1.0-py3-none-any.whl Python 3 none any Details for evalkit_harness-0.1.0-py3-none-any.whl

Total release size: 438.4 kB

Release files / evalkit_harness-0.1.0.tar.gz

Download URL evalkit_harness-0.1.0.tar.gz
Size 190.2 kB
Tags Source
SHA-256 checksum
How to use checksums
6066685e09b63e483debe7df37548cfd2108a0b499365429e1c2ba78b6c9ee89
BLAKE2b-256 checksum
How to use checksums
c9757a4ae643a1b7ebd5388e08f137f6afebf1a658bfd23887453ffddf4a9951
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / evalkit_harness-0.1.0-py3-none-any.whl

Download URL evalkit_harness-0.1.0-py3-none-any.whl
Size 248.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8921677faf675e0b5ce95962e6d2644f833e51da8c70ff27997abec772bbcc2d
BLAKE2b-256 checksum
How to use checksums
66a9aa6e1f70009b03bcb5658a3b32206da8cb5a705a6e353bf41699b3607411
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for evalkit-harness 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page