judge-audit

Independent calibration audits for AI judges. When a judge says 90 %, is it right 90 % of the time?

Everyone is shipping judgment models — TypeSafe's Jev, LLM-as-judge, guardrails, routers — and every one of them returns a confidence. Nobody checks whether that number means anything. judge-audit runs any judge in shadow mode against decisions your humans already made and answers the four questions that matter before you automate:

pip install kunko-judge-audit           # every release ships Sigstore-signed build provenance
judge-audit run examples/email-routing/labels.jsonl --judge simulated   # no API key needed
judge-audit check examples/email-routing/labels.jsonl --judge simulated --baseline audit-result.json

▶ 24-second launch video · vertical cut

simulated is a seeded simulator so you can see the whole pipeline in ten seconds; every report it touches is stamped SIMULATED. To audit a real vendor, see docs/real-audits.md.

The first independent audits of Jev

Same judge (TypeSafe Jev, via Vercel AI Gateway), three jobs, every raw response committed under docs/runs/ so anyone can recompute every number (python scripts/verify_published.py does, in CI).

What we would tell a client. The email numbers are the vendor's story and they hold up, including under attack. The router numbers are the buyer's story: the first prompt anyone would write routed every hard task to the cheap model at 96 % median confidence, and its 66.7 % accuracy is exactly what a coin glued to "easy" scores; two descriptive sentences took the same judge to 97.5 % with confidence that finally means something. None of that is visible from a benchmark leaderboard; all of it is visible from a calibration audit on your own decisions.

Honest limits: every dataset is synthetic and seeded (generators in examples/); n is small; ground truth for routing is by construction, not by running the cheap model. Read the Caveats section of each report before quoting it.

How it works

1. Labeled dataset — JSONL rows {state, questions, labels}. Your humans already decided; the judge must reproduce them.
2. Shadow run — the judge scores every row. Nothing is automated; everything is recorded: decision, confidence, latency, cost, raw response.
3. Audit report — calibration (ECE, reliability bins), selective prediction (accuracy at every coverage level), cost, latency percentiles, plus provenance: model, backend, timestamp, dataset hash.
4. CI gate — judge-audit check --baseline baseline.json --max-ece-drift 0.02 fails the build when the judge degrades.

Exit codes: 0 ok · 1 drift detected · 2 usage or configuration error (the message says what to fix).

In CI: the GitHub Action runs the audit on every push or pull request and fails the build on drift:

- uses: kunko-ai-labs/judge-audit@v0.3      # or pin the release's commit SHA
  with: { labels: audits/labels.jsonl, judge: jev, baseline: audits/baseline.json }
  env: { AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} }

One sticky PR comment with the audit table (n, accuracy, ECE, zero-error coverage, cost, p99, verdict vs baseline), the report in the job summary, evidence as an artifact. Outputs accuracy, ece, zero-error-coverage, drift for anything downstream.

From inside an agent: pip install "kunko-judge-audit[mcp]" then claude mcp add judge-audit -- judge-audit-mcp (or the equivalent in Cursor). The agent gets run_audit, check_drift and list_judges and can audit the judge it is about to rely on without leaving the session. See docs/integrations.md.

Judge interface

Anything that maps (state, questions) -> (decision, confidence) plugs in:

from judge_audit import Judge, Question, Judgment

class MyJudge(Judge):
    name = "my-judge"

    def decide(self, state: str, questions: list[Question]) -> list[Judgment]:
        ...  # return Judgment(question=q.name, decision="spam", confidence=0.93)

Ships with three: jev (TypeSafe Jev — and, via JEV_ENDPOINT, any Jev-compatible server such as OpenJev), llm (any chat model with a confidence prompt: Claude through the official SDK, or anything OpenAI-compatible — OpenAI, Ollama, vLLM) and simulated. Run the same dataset through several and you have the first row of the Judge Arena. Details in docs/judges.md.

Why calibration, not accuracy

Accuracy tells you who wins a benchmark. Calibration tells you what you can automate safely. A 96 %-accurate judge that is confident on the 4 % it gets wrong is a liability; a 90 %-accurate judge that flags its own doubt is an asset. Audit the honesty, not the trophy.

What lives where

Roadmap

Score questions + MCE (the regulator's number) → Judge Arena (Jev vs OpenJev vs Claude vs open models on the same datasets, published) → AI Act evidence dossier. Details and reasons in docs/ROADMAP.md; the live backlog is the issues.

FAQ

Is this a benchmark? No. A benchmark ranks judges on a fixed test. An audit checks one judge on your decisions and tells you what you can automate. The datasets here are worked examples, not a leaderboard — yet.

Why is Jev the first judge? It is the first judgment model sold on calibrated confidence as the headline feature. A claim that specific deserves an independent check.

Can I trust a 100 % result? Only as far as the dataset. The clean-email audit says the judge handles templated business email; it says nothing about your inbox. That is why the adversarial and routing audits exist.

Does it send my data anywhere? Only to the judge endpoint you configure. Reports and checkpoints are local files; commit them or not.

License

Apache-2.0 — see LICENSE. Sister project: agent-assurance (deterministic checks on what an agent may do; this repo is the probabilistic half: whether its judgment can be trusted).