evalgate
Decide whether an eval delta is a real regression or just sampling noise, and fail CI only when it is real.
A model eval that drops from 90.0% to 89.4% on a 1,000-example set looks like a
regression, but on that sample size it is noise. Gating CI on the raw number
makes the build flap; ignoring it lets real regressions through. evalgate
runs the appropriate significance test and fails only when the candidate is
significantly worse.
$ evalgate proportions \
--baseline-score 0.900 --baseline-n 1000 \
--candidate-score 0.894 --candidate-n 1000
verdict worse, but within noise
difference -0.0060
p-value 0.6232
alpha 0.05
# exit code 0 -> build passes
Install
$ pip install evalgate-cli # from PyPI, once released
$ pip install git+https://github.com/jmweb-org/evalgate # latest, available now
Pure standard library plus typer and rich. No heavy dependencies.
Usage
From two aggregate accuracies
$ evalgate proportions \
--baseline-score 0.90 --baseline-n 2000 \
--candidate-score 0.87 --candidate-n 2000 \
--alpha 0.05
Uses a two-proportion z-test on the accuracies and their sample sizes.
From paired per-example results
When both models were evaluated on the same examples, a paired test is more powerful. Give a CSV with per-example correctness (or predictions plus a truth column):
$ evalgate paired results.csv --baseline base_correct --candidate cand_correct
$ evalgate paired results.csv --baseline pred_a --candidate pred_b --truth label
Uses McNemar's test: an exact binomial test on the discordant pairs, or the continuity-corrected chi-squared approximation for large samples.
In CI
- run: evalgate proportions --baseline-score 0.90 --baseline-n 2000
--candidate-score "$SCORE" --candidate-n 2000
Verdicts and exit codes
| Verdict | Meaning | Exit |
|---|---|---|
improvement |
Candidate is better | 0 |
unchanged |
No measurable difference | 0 |
noise |
Worse, but not significant at alpha |
0 |
regression |
Significantly worse | 1 |
A bad invocation (scores out of range, missing column, unreadable file) exits 2.
JSON output
When using the --json flag, evalgate returns a JSON object with the following fields:
| Field | Type | Description |
|---|---|---|
verdict |
string |
One of improvement, unchanged, noise, or regression. |
p_value |
float |
The p-value from the statistical test. |
difference |
float |
Signed difference between candidate and baseline. A negative value means the candidate performed worse. |
alpha |
float |
Significance threshold used for the test. |
is_regression |
boolean |
true if the result is a statistically significant regression; otherwise false. |
test |
string |
Statistical test used (two_proportion_z or mcnemar). |
baseline_only_correct |
integer |
(Paired mode only) Number of examples only the baseline answered correctly. |
candidate_only_correct |
integer |
(Paired mode only) Number of examples only the candidate answered correctly. |
What it does and does not do
It answers one question: is this difference larger than sampling variation?
It does not correct for multiple comparisons across many evals, and a
noise verdict means "not proven", not "proven equal". For small evals,
collect more examples rather than trusting a borderline p-value.
License
MIT. See LICENSE.
Metadata
Release files for evalgate-cli 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evalgate_cli-0.2.0.tar.gz | 11.1 kB | Without provenance | Details for evalgate_cli-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evalgate_cli-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for evalgate_cli-0.2.0-py3-none-any.whl |
Total release size: 21.9 kB
Release files / evalgate_cli-0.2.0.tar.gz
| Download URL | evalgate_cli-0.2.0.tar.gz |
|---|---|
| Size | 11.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ec1fa4641f7f610db3ac51fd9d0eb28cf619f4dbe89ac629f846fc05f2e7241e
|
|
BLAKE2b-256 checksum How to use checksums |
63da975a08602af0318babb16d15025b9ab9013ccbf6e4a2ef121926535c094e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.20 {"installer":{"name":"uv","version":"0.11.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / evalgate_cli-0.2.0-py3-none-any.whl
| Download URL | evalgate_cli-0.2.0-py3-none-any.whl |
|---|---|
| Size | 10.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ab11bf804d4ed588180e95384125c0b97baf8043ca1abdaa84ee1fcf169cf731
|
|
BLAKE2b-256 checksum How to use checksums |
d13da5459a86750e6979f2116933c3626237c341cdc27ad8742cad3142cee96f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.20 {"installer":{"name":"uv","version":"0.11.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for evalgate-cli 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.