This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 1.3.0 instead.
Reason given by maintainers: Old version with a security issue (GHSA-5xgq-4r93-29m3). Use the newest judgekeeper
judgekeeper ™
Check your LLM-as-a-judge in 5 steps.
Find. Mark. Measure. Fix. Re-check.
An LLM-as-a-judge is an AI model that grades another model's answers. judgekeeper shows how often your LLM-as-a-judge agrees with you, and where it goes wrong.
Install
In your project folder, in its virtual environment:
pip install judgekeeper
judgekeeper
In a uv virtual environment, install it with uv pip install judgekeeper instead.
With no virtual environment, run uvx judgekeeper. judgekeeper runs on Python 3.11 to 3.14.
Reads promptfoo, DeepEval, Inspect AI, MLflow, or your own CSV, TSV or JSONL file. In your own file, judgekeeper looks for three columns: input (the question), output (the answer) and verdict (Pass or Fail from your LLM-as-a-judge). For Inspect AI logs or MLflow: pip install "judgekeeper[inspect]" or pip install "judgekeeper[mlflow]" (with uv: uvx --from "judgekeeper[inspect]" judgekeeper or uvx --from "judgekeeper[mlflow]" judgekeeper).
Using a coding agent? Give it this page. It runs the steps and gives you the marking page link; you mark, then it explains the result.
The 5 steps
- Find:
judgekeeperfinds your eval results. - Mark: a page opens in your browser. Mark answers Pass or Fail; the LLM-as-a-judge's grade stays hidden. "See the numbers so far" appears with your first numbers, and "See your result" once there is a result.
- Now and then an answer you marked comes back once, to see how steady your marks are. Mark it as you see it.
- No browser here, or a coding agent runs it?
judgekeeper --no-browserprints the link to open instead. - Already have Pass/Fail labels? Run
judgekeeper --labels your-labels.csv(orjudgekeeper --labelsfor labels already in your results), then mark more if needed.
- Measure: run
judgekeeperagain to see how often it agrees with you.- A result needs at least 15 Pass and 15 Fail marks from you. It also needs 20 marks on answers your LLM-as-a-judge passed and 20 on answers it failed (or all of them, if there are fewer).
- If your LLM-as-a-judge gives scores, a plain
judgekeeperasks for the pass mark again, with the one you gave as the default. With no terminal, it prints the--pass-ifto give.
- Fix:
judgekeeper fixhelps you write a better rule and tests it on marks kept aside.- It is a loop: change the rule, grade again, then run
judgekeeper fix <new results file>. - With promptfoo, the rule to change is usually in
.judgekeeper/fix/regrade.yaml. Whenfixcan't write that file, it says how to grade again. - With few marks, "can't tell yet" is a normal answer.
- It is a loop: change the rule, grade again, then run
- Re-check: after you change the model or the rule, grade your marked answers again, then run
judgekeeper check <new results file>.- Your marked answers are in
.judgekeeper/fix/marked-answers.csv. With promptfoo,fixusually writes.judgekeeper/fix/regrade.yamlto grade them. - Grading only your marked answers is enough to compare the two. To count new marks, grade all your answers with your new LLM-as-a-judge.
- If you switched to a new pass mark with
judgekeeper check --pass-if, a plainjudgekeeperandjudgekeeper markask for the pass mark again, with the new one as the default.
- Your marked answers are in
In your project, judgekeeper reads your results files and writes only in .judgekeeper/, plus judgekeeper.toml after you say yes. It makes no AI call and needs no API key. See SECURITY.md.
The numbers
- TPR: of the answers you passed, how many it passed too. The result page calls it "When you say Pass, your LLM-as-a-judge also says Pass".
- TNR: of the answers you failed, how many it failed too. The result page calls it "When you say Fail, your LLM-as-a-judge also says Fail".
- Kappa: how much you agree beyond what matching at random would give.
TPR and TNR each come with a range, such as "about 90% (between 85% and 93%)" (when you mark every answer, the numbers have no range). The result says "good enough to use" only when a wider range (99%) starts at 80% or more for both. That way you can look after every mark. The numbers are about the answers in your results, worked out from the ones you marked. They are not a promise about new answers.
How the numbers are worked out: reference.
Checked on public data
These runs use public datasets that people already labelled. One adds made-up empty answers that should fail, and one is a simulation. Where the true number is known, they test whether judgekeeper's ranges hold it. They also show what judgekeeper says about real LLM-as-a-judge setups. Each line links its report.
- JailbreakBench: the ranges held the true values in 96 of 100 random tests with 100 marks each (TPR, TNR and kappa at once; another 100 tests give a slightly different count).
- RAGTruth: news summaries. Three LLM-as-a-judge setups, 3 runs each, and 3 more runs at a lower effort setting: needs work in all 12. Changing the model did not close the gap.
- RAGTruth question answers and data-to-text: "needs work" in all 4 runs.
- JailbreakBench graded by two other setups: "needs work" for both; each passed more than half of the expert-failed answers it graded (setup B could not grade 61 of 300).
- Empty answers: one model failed 0 of 20 empty summaries that should fail.
- Simulation: the ranges held the true value 96.7 times in 100 on average.
More
A guide on the website walks through the 5 steps. judgekeeper --help lists every command, and the reference explains each one. Made by one person; issues welcome, pull requests are not taken.
License
judgekeeper and the judgekeeper logo are trademarks of Sathvik Thota. The Apache-2.0 license covers the code, not the name or logo.
Metadata
Release files for judgekeeper 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| judgekeeper-1.1.0.tar.gz | 544.9 kB | With provenance | Details for judgekeeper-1.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| judgekeeper-1.1.0-py3-none-any.whl | Python 3 | none | any | Details for judgekeeper-1.1.0-py3-none-any.whl |
Total release size: 1.1 MB
Release files / judgekeeper-1.1.0.tar.gz
| Download URL | judgekeeper-1.1.0.tar.gz |
|---|---|
| Size | 544.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1117e78dc293449a323ade01abb573cdf3987b29e4b62458a57b74119eb7cb45
|
|
BLAKE2b-256 checksum How to use checksums |
bd7044b77ba0b1220869af055ddf684a40a5b426239544a4994181415bc5d91d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logRelease files / judgekeeper-1.1.0-py3-none-any.whl
| Download URL | judgekeeper-1.1.0-py3-none-any.whl |
|---|---|
| Size | 572.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
421d70b916777e22d67af3934c3845fa64babe7862e10a09381018fa974006e6
|
|
BLAKE2b-256 checksum How to use checksums |
33ff6bef8cb5e817800554becc6efad32ee95f129eed3c8d07b881f46e9f63f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logSecurity for judgekeeper 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for judgekeeper-1.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.