EvalKit
On 200 PubMedQA questions, 4 models (claude-sonnet-5-5, claude-haiku-5-5, gpt-6.1-sol, gpt-6-luna) answered from the abstract and cited its sentences; two judges from different families (claude-opus-5-5 and gpt-6.1-sol) checked every claim. The most grounded model was gpt-6.1-sol, correct and fully grounded on 57.5% (95% CI 50.5%–64.0%) by claude-opus-5-5, 46.0% (95% CI 39.0%–53.0%) by gpt-6.1-sol; it is separated from every other model after a Holm correction (gpt-6.1-sol also judged its own answers; claude-opus-5-5 agrees); the most accurate model was claude-sonnet-5-5 (77.0% correct labels); the headline also requires every claim to be supported by the sentences the answer cites. Caveats: judge not calibrated for this task (see docs/calibrate-your-task.md). Treat these numbers as provisional.
Full report · results in docs/results/
pip install 'evalkit-harness[anthropic,openai]'
evalkit init # scaffold a project and a three-case sample task
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 --dry-run # estimate
evalkit run evalkit.yaml --task tasks/my_task.yaml --n 3 # needs keys in .env
Questions from PubMedQA (Jin et al., 2019, MIT licence); see DATA_CARD.md. Research benchmark, not medical advice.
Evaluate your own task across Claude and OpenAI models: accuracy, groundedness, citation validity, cost and latency side by side, with confidence intervals and two cross-family judges whose error rate is measured against held-out hand labels.
Status: v0.1.0, the first public release. To work on EvalKit itself, clone it and run
uv sync --all-extras.
Start here: Quickstart (a first result in under five minutes) · Write your own task (JSONL or CSV) · Control cost · Calibrate the judges · Scorers and adapters · Run it in CI · Versioning
Built-in tasks: pubmedqa (yes / no / maybe research questions) and scifact (supports / refutes / not enough info for scientific claims); see DATA_CARD.md.
No telemetry: data goes only to the providers you configure (PRIVACY.md). Security and key handling: SECURITY.md. Contributions welcome: CONTRIBUTING.md.
The package is published as evalkit-harness (the name evalkit on PyPI belongs to an unrelated project). The command and the import are both evalkit.
- Product: EvalKit PRD
- Design: EvalKit Tech Spec
- Build status:
uv run python scripts/status.py
MIT licence.
Metadata
Release files for evalkit-harness 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evalkit_harness-0.1.0.tar.gz | 190.2 kB | Without provenance | Details for evalkit_harness-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evalkit_harness-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for evalkit_harness-0.1.0-py3-none-any.whl |
Total release size: 438.4 kB
Release files / evalkit_harness-0.1.0.tar.gz
| Download URL | evalkit_harness-0.1.0.tar.gz |
|---|---|
| Size | 190.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6066685e09b63e483debe7df37548cfd2108a0b499365429e1c2ba78b6c9ee89
|
|
BLAKE2b-256 checksum How to use checksums |
c9757a4ae643a1b7ebd5388e08f137f6afebf1a658bfd23887453ffddf4a9951
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / evalkit_harness-0.1.0-py3-none-any.whl
| Download URL | evalkit_harness-0.1.0-py3-none-any.whl |
|---|---|
| Size | 248.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8921677faf675e0b5ce95962e6d2644f833e51da8c70ff27997abec772bbcc2d
|
|
BLAKE2b-256 checksum How to use checksums |
66a9aa6e1f70009b03bcb5658a3b32206da8cb5a705a6e353bf41699b3607411
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for evalkit-harness 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.