judgekeeper

Check your LLM-as-a-judge.

See how often it agrees with human labels, in one command.

An LLM-as-a-judge is an AI model that grades the answers of another AI model. Teams use one because reading every answer by hand is slow.

Use it on your own judge

cd your-project
uvx judgekeeper@1.0.0b1

You need Python 3.11 or newer and uv. The version is in the command because uv and pip skip betas unless you name one. Without uv, install it in your project's own Python environment, then run it:

pip install judgekeeper==1.0.0b1
judgekeeper

• It reads the results your eval tool already saved (promptfoo, DeepEval, Inspect AI, MLflow, or a CSV) and checks your judge's verdicts for free.
• It opens a page in your browser. You mark answers Pass or Fail without seeing what your judge said.
• Run it again the same way when you are done: it shows how often your judge agrees with you.
• No AI calls and no API key. judgekeeper never runs your app.

This is a beta: fixing your judge and re-checking it on its own are coming.

With a coding agent

Paste this into Claude Code, Codex or Cursor:

Run `uvx judgekeeper@1.0.0b1` in this project and follow what it says.

The agent runs judgekeeper and tells you when the page is open; you do the marking. Its rules are in one file, skills/judgekeeper/SKILL.md.

What you get

TPR: of the answers you passed, how many your judge passed. TNR: of the answers you failed, how many it failed. Kappa: how much you agree beyond chance. Each comes with a range, and judgekeeper says the smallest drop your marks can see. The numbers are estimates from the answers you marked, not a promise about the next ones.

Numbers

• Its ranges hold on real data. We tested the ranges for how often the judge passed answers people passed (TPR) and how often it failed answers people failed (TNR). We used 5 real judges from 3 public datasets (LLMBar, MT-Bench, LLMJudge), with 50 human labels per check, over 1,000 checks per judge. On average the value from all the human labels fell inside the range in 96.0 checks in 100 (lowest 94.6, highest 98.4). Real-data check
• One of those checks: GPT-4 as the judge on MT-Bench, 50 labels: TPR 0.82 (range 0.71 to 0.91), and the value from all 1,814 human labels is 0.85; TNR 0.49 (range 0.36 to 0.65), and from all labels 0.56.
• In simulation too: the ranges held the true value at least 93.6 times in 100 in every tested case, and 96.2 on average. Coverage check
• A real judge, checked: Claude Haiku 4.5 on LLMBar, 419 items: TPR 0.95, TNR 0.88, kappa (agreement after taking away lucky guesses) 0.83. Report
• Tested on Python 3.11 to 3.14 on Linux, and on Windows. Test runs

Status

Beta (1.0.0b1): commands and report fields may still change. judgekeeper --help lists every flag, and the changelog says what changed.

Issues

This project is maintained by one person and does not accept pull requests. Bug reports and ideas are welcome as issues.

License

MIT, for the code. The name judgekeeper and the logo are not covered: see TRADEMARKS.md.