Skip to main content
Yanked

This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 1.3.0 instead.
Reason given by maintainers: Old version with a security issue (GHSA-5xgq-4r93-29m3). Use the newest judgekeeper.

judgekeeper logo

judgekeeper

Check your LLM-as-a-judge in 5 steps.
Find. Mark. Measure. Fix. Re-check.

PyPI version Tests Apache-2.0 license

Website · Guide · Reference · Changelog

An LLM-as-a-judge is an AI model that grades another model's answers. judgekeeper shows how often yours agrees with you, and where it goes wrong.

judgekeeper finds the results, opens the marking page, and shows how often the LLM-as-a-judge agrees with you.

Install

In your project folder, in its virtual environment:

pip install judgekeeper
judgekeeper

Or, with no virtual environment: uvx judgekeeper. Python 3.11 to 3.14. Reads promptfoo, DeepEval, Inspect AI, MLflow, or your own CSV, TSV or JSONL file. For Inspect AI logs or MLflow: pip install "judgekeeper[inspect]" or pip install "judgekeeper[mlflow]".

The 5 steps

  1. Find: judgekeeper finds your eval results.
  2. Mark: a page opens in your browser. Mark answers Pass or Fail; the LLM-as-a-judge's grade stays hidden.
  3. Measure: run judgekeeper again to see how often it agrees with you. You need at least 15 Pass and 15 Fail marks.
  4. Fix: judgekeeper fix helps you write a better rule and tests it on marks kept aside.
  5. Re-check: judgekeeper check shows if it still agrees after you change the model or the rule.

judgekeeper only reads files. It makes no AI call and needs no API key. See SECURITY.md.

The numbers

  • TPR: of the answers you passed, how many it passed too.
  • TNR: of the answers you failed, how many it failed too.
  • Kappa: how much you agree beyond what matching at random would give.

Each comes with a range, such as "about 90% (between 84% and 93%)". The result says "good enough to use" only when the low end of both TPR and TNR is at least 80%. The numbers describe the answers you marked, not a promise about the next ones.

How the numbers are worked out: reference.

Checked on public data

  • JailbreakBench: the ranges held the true value in 93 of 100 tests (all three numbers at once).
  • RAGTruth: three LLM-as-a-judge setups on news summaries: judgekeeper said "needs work" in all 12 runs. Changing the model did not close the gap.
  • RAGTruth question answers and data-to-text: "needs work" in all 4 runs.
  • JailbreakBench graded by two other setups: "needs work" for both; each passed more than half of the answers the experts failed.
  • Empty answers: one model failed 0 of 20 empty summaries that should fail.
  • Simulation: the ranges held the true value 96.2 times in 100 on average.

More

A guide on the website walks through the 5 steps. judgekeeper --help lists every command, and the reference explains each one. Made by one person; issues welcome, pull requests are not taken.

License

Apache-2.0 for the code. The name and logo are not covered: see TRADEMARKS.md.

Metadata

Release files for judgekeeper 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for judgekeeper 1.0.0
File Size Uploaded Provenance Actions
judgekeeper-1.0.0.tar.gz 452.6 kB With provenance Details for judgekeeper-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for judgekeeper 1.0.0
File Interpreter ABI Platform Actions
judgekeeper-1.0.0-py3-none-any.whl Python 3 none any Details for judgekeeper-1.0.0-py3-none-any.whl

Total release size: 932.2 kB

Release files / judgekeeper-1.0.0.tar.gz

Download URL judgekeeper-1.0.0.tar.gz
Size 452.6 kB
Tags Source
SHA-256 checksum
How to use checksums
101590375c8c1bbf5f333f26c7405ec326de5131614a97c87b0ec8dbc0726328
BLAKE2b-256 checksum
How to use checksums
130b9ccfb6033c052cdb40aecde0cf60e45950feb462e4090cd9f8c50bf00283
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Release files / judgekeeper-1.0.0-py3-none-any.whl

Download URL judgekeeper-1.0.0-py3-none-any.whl
Size 479.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b769a59408d2826d45ef47b04da3ac03c238e9094d9c1510cb0109d1210b3091
BLAKE2b-256 checksum
How to use checksums
85b7eb785e4c4f44ce89d50049627b98fec23eafca7ab51f9b24a24d77ec17d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Security for judgekeeper 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for judgekeeper-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page