judgekeeper

Check your LLM-as-a-judge.

See how often it agrees with human labels, in one command.

An LLM-as-a-judge is an AI model that grades the answers of another AI model. Teams use one because reading every answer by hand is slow.

Try it

pip install judgekeeper
judgekeeper demo

It needs no key and no data. The demo checks a recorded judge on answers that ship with the package. Without installing: uvx judgekeeper demo. On a Mac the command is pip3. If the install is refused, see Install.

Use it on your own judge

judgekeeper check results.csv --judge verdict --human label

The table has one row per answer: what the judge said and what a person said.

What you get

judgekeeper demo: a recorded judge on 100 synthetic single-output items, 3 runs.
Of the answers people passed, the judge passed 97%. Of the answers people failed, the judge failed 86%. It is usable with care.
TPR 0.97 (95% CI 0.88 to 0.99), TNR 0.86 (95% CI 0.73 to 0.94), kappa 0.84
Usable with care: TPR 0.97, TNR 0.86, kappa 0.84 against human labels. Read the flags below before relying on it.
  - TNR is 0.86 (between 0.80 and 0.90): usable with care.
open judgekeeper-demo/report.html (data in judgekeeper-demo/report.json)

The second line is the answer in plain words. The next two lines give the same result as numbers; How it works says how to read them. The indented line is a flag: what to look at before you rely on this judge. The last line is a report page you can open.

A report from a real judge on a public dataset: LLMBar judged by Claude Haiku 4.5. Three worked examples with two judges each: See it work.

When you need more

Status

Alpha (0.1.3): commands and report fields may still change. Use it on your app has the four steps, How it works has the background, the reference has every command and flag, and the changelog says what changed.

License

MIT.