Skip to main content
judge-audit — calibration audits for AI judges

CI Release License: Apache-2.0 Python 3.10+

Can your AI judge decide alone?

judge-audit replays decisions your people already made and tells you how much of that work an AI judge can take over, at the error you accept, with the bound attached. Open source, runs in CI, no LLM inside the measurement, and every published number recomputes from committed evidence.

What we contribute.

  • An independent audit, not a model. judge-audit measures the confidence an AI judge declares against decisions people already made. It judges nothing itself.
  • Evidence anyone can recompute. Every published figure regenerates from committed checkpoints, and CI fails when a report or this README drifts from its evidence (scripts/verify_published.py, scripts/verify_readme.py).
  • Pre-registered studies, separate reviews. The v0.5 and v0.6 studies were pre-registered before their first model call (v0.6: the plan's sha256 committed at the freeze; amendments A1–A4 hashed before the calls or the analysis they govern, A5, an erratum on wording, at publication). Each pull request goes through a separate review and QA pass before it merges, by AI review agents that did not write the change, with the verdicts posted on each pull request. No external human reviewer has read the study plans.
  • Our own security advisories. We publish the weaknesses we find in our own tools: GHSA-52xj-j738-jpm3 (judge-audit: an API key in error output, or sent on a redirect) and GHSA-f7jx-f5h6-pmx7 (agent-assurance).

judge-audit in 25 seconds: the question, the audit run, the share each judge can decide alone on 3,080 real banking queries, the CI gate failing below the minimum

🎯 Safe automation rate The share of decisions a judge can take alone at the error you accept (1, 2, 5, 10 %), the confidence threshold to deploy, and an out-of-sample check.
🌡️ Calibration When it says 90 %, is it right 90 % of the time? ECE, Brier score, reliability diagram, with intervals.
🚦 CI gate judge-audit check fails the build when the judge drifts or the rate falls below your minimum.
🤖 Where you work CLI, GitHub Action with a PR comment, and an MCP server for Claude Code, Cursor or any agent.

Quickstart · What you get · v0.6 findings · v0.5 findings · Judges supported · ▶ launch video

v0.6 findings

v0.6 in short. 12 AI judges, the same 2,974 items of MASSIVE 1.1 in English, Spanish and Catalan, pre-registered before the first model call (plan, sha256 committed at the freeze; amendments A1–A5). Six pre-registered transfer tests asked whether a threshold validated on English rows at a 5 % error target lets more than 5 % errors through on the same items in Spanish or Catalan: none was resolved at its Holm-corrected level, so no breach was shown; that is not a pass: the approximate one-sided 95 % upper bounds on the pooled target-language error run from 3.30 % (T6) to 8.65 % (T2), above 5 % in 5 of 6 tests, and 3 of the 6 came out as predicted. Demonstrated on this sample, and only on it (MASSIVE is translated public data, probably seen in pretraining): declared confidence does not always match accuracy; mean confidence minus accuracy is +0.080 to +0.085 for jev, −0.052 to −0.067 for gpt-6-luna and +0.063 to +0.099 for claude-sonnet-4.5 across the three languages. The differences between languages are 95 % intervals with no correction for multiplicity, so they are not called demonstrated (§2). The number to act on is the same audit on your own decisions.

At a glance: v0.6, MASSIVE 1.1, 2,974 translated public items in English, Spanish and Catalan (not your traffic), 12 judges: the held-out share automated by thresholds chosen at a 5 % error target (the threshold chosen on one half of the items, applied to the other), with its approximate bound; met when that bound is within 5 %, at the study's split (seed 2026).

Met is per split. gpt-6-luna en-US is met at the study's split (seed 2026), and its share moves over split seeds 2027–2036 (21.8–39.4 %); on another split a met cell can miss the target. The bounds of the other splits are not computed here.

Judge en-US es-ES ca-ES
jev 29.0 % (bound 5.8 %, not met; splits 30.7–55.6 %) 25.3 % (bound 7.4 %, not met; splits 0.0–23.6 %) none (splits 0.0–19.5 %)
gpt-6-luna 18.2 % (bound 4.6 %, met; splits 21.8–39.4 %) none (splits 0.0–13.3 %) 5.3 % (bound 9.8 %, not met; splits 0.0–0.0 %)
claude-sonnet-4.5 21.5 % (bound 2.9 %, met; splits 21.5–21.5 %) 8.5 % (bound 5.7 %, not met; splits 0.0–8.8 %) none (splits 0.0–6.9 %)
clef-flash (exploratory) 40.6 % (bound 5.0 %, met; splits 0.0–44.4 %) 38.5 % (bound 5.7 %, not met; splits 4.0–44.2 %) 39.0 % (bound 4.2 %, met; splits 18.0–60.0 %)
decider-2b (exploratory) 31.1 % (bound 7.7 %, not met; splits 0.0–9.7 %) 10.4 % (bound 7.4 %, not met; splits 0.0–4.5 %) none (splits 0.0–0.0 %)
decision2-kai-0.6b (exploratory) 4.1 % (bound 7.3 %, not met; splits 0.0–7.4 %) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %)
deepseek-r1 (exploratory) none (splits 0.0–28.7 %) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %)
gemini-3.6-flash (exploratory) 55.1 % (bound 3.4 %, met; splits 55.1–55.1 %) 26.6 % (bound 5.3 %, not met; splits 26.0–52.8 %) 27.1 % (bound 5.1 %, not met; splits 44.6–53.6 %)
laya-multilingual (exploratory) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %)
microsoft-decision-1 (exploratory) 80.7 % (bound 4.6 %, met; splits 80.2–82.7 %) 72.3 % (bound 4.6 %, met; splits 40.5–73.7 %) 30.1 % (bound 4.9 %, met; splits 31.5–60.6 %)
strands-decider-2b (exploratory) 3.9 % (bound 14.4 %, not met; splits 0.0–0.0 %) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %)
unsloth-decision-4b (exploratory) none (splits 0.0–4.8 %) none (splits 0.0–0.0 %) none (splits 0.0–0.0 %)

Bound: the one-sided 95 % upper bound on the held-out errors of both halves pooled, rounded up; approximate, since each half's threshold depends on the other half; computed after the study from its published counts, not pre-registered (#213). Met: that bound is within 5 %; only a share that meets it is called validated. Each met cell is a separate statement, each at 95 %, approximate; read together they do not all hold at 95 %, so no claim is made about all of them at once, and none should be picked as the best. None: no threshold was validated on either half at seed 2026. Splits: the same share over split seeds 2027–2036; that range need not contain the seed-2026 share, and a share far outside it rests on one split. Labels are carried over by localisation, and no share says anything about future traffic; judges marked exploratory are in no pre-registered test. No ranking: rows follow the study's roster. The shares, their ranges, every other figure with its intervals and the cost of each run are in the v0.6 findings in full; the bounds are computed here from their counts.

v0.5 findings

In short. Asking Qwen3-8B and gemini-3.6-flash several times and counting the votes ranked their errors worse than their own verbalized number in 3 of 4 tests under the pre-registered rule (4 of 4 under the re-reading), and one verdict (H2 for gemini-3.6-flash) changes with how an answer that copies an option's description is scored. On BANKING77 at a 5 % error target, validated on held-out texts (approximate), Jev native probability can decide 41.0 % (27.0 %–45.1 % over split seeds 2027–2036) of the texts alone, gemini-3.6-flash verbalized 14.3 % (7.6 %–30.0 %), Qwen3-8B token log-probability none (0.0 %–4.4 %). Whether the bound is within the target depends on the split: at split seed 2026 Jev's 41.0 % has a bound of 3.9 %, within 5 %; on another split of the same checkpoints (split seeds 2033 and 2036) the same judge at the same 5 % target automates fewer texts with a bound above 5 %, not met; the range over split seeds shows how far the share moves. Caveats: public datasets, probably seen in pretraining; one run each, except Jev and Qwen3-8B verbalized on BANKING77 (3 runs); label noise not measured; the held-out slice not run (#106). The number to act on is the same audit on your own decisions.

At a glance: BANKING77, 3,080 human-labelled banking queries: the share each judge decides alone at a 5 % error target, validated on held-out texts (approximate) at split seed 2026; another split can change the result.

Judge and confidence Validated share at a 5 % error target (approximate) 95 % upper bound on the error rate Range over split seeds At a 10 % error target
Jev, native probability 41.0 % 3.9 % 27.0–45.1 % 72.3 % (bound 9.2 %)
gemini-3.6-flash, verbalized 14.3 % 2.7 % 7.6–30.0 % 45.5 % (bound 5.3 %)
Qwen3-8B, token log-probability none — 0.0–4.4 % 12.5 % (bound 7.2 %)

Validated: the threshold is chosen on one half of the texts and applied to the other, both ways, and the 95 % upper bound on the held-out error rate is within the target. The bound is the one-sided Clopper–Pearson bound on the held-out errors of both halves pooled, rounded up; it treats the two held-out halves as independent draws, an approximation since each half's threshold depends on the other half. A share observed on all rows without a bound is not a basis for deployment.

Teams are shipping judgment models — TypeSafe's Jev, LLM-as-judge, guardrails, routers — that return a confidence with each decision. The literature studies calibration (Guo et al. 2017; Shao 2026; Huang et al. 2026); what a team needs before automating is the same measurement on its own decisions. judge-audit runs any judge in shadow mode against decisions your humans already made and answers the four questions that matter before you automate:

Question Metric Why a buyer cares
When it says 80 % confident, is it right 80 % of the time? ECE, reliability diagram A confident-and-wrong judge automates its own mistakes
What share of the work can it decide alone at a bounded error? safe automation rate, accuracy-coverage curve The ROI number — with its error bound, the threshold to deploy, and the labels behind it
What does it really cost, and how bad is the latency tail? $ per decision, p50 / p99 The demo is cheap; the tail is what pages you
Has it drifted since last week? judge-audit check CI gate Vendors update models without telling you

The study's design, the eight confirmatory tests and every caveat: v0.5 findings in full.

Quickstart

pip install kunko-judge-audit           # every release ships Sigstore-signed build provenance
judge-audit run examples/email-routing/labels.jsonl --judge simulated --target 0.10   # no API key needed
judge-audit check examples/email-routing/labels.jsonl --judge simulated --baseline audit-result.json --target 0.10 --min-safe-rate 0.10:0.60

The start of the run line, SIMULATED:

SIMULATED — not a real vendor audit · judge=simulated n=200 accuracy=85.5% [75.1, 93.7] safe_automation@10%=not_met(75.0% bound_95_heldout_approx=16.6%) all_rows=65.5% …

simulated is a seeded simulator so you can see the whole pipeline in ten seconds; every report it touches is stamped SIMULATED, and its figures say nothing about any real judge. safe_automation@10% leads with the held-out check at a 10 % error target: not_met, because the one-sided 95 % upper bound on the held-out error rate (16.6 %, approximate: it pools the two held-out halves) is above the target, so the 75.0 % held out is not validated. A share is shown as validated only when that bound is within the target. all_rows (65.5 %) is the share the threshold to deploy covers on every labelled decision, the share check --min-safe-rate gates. At the default target, 5 %, the simulator reaches no threshold and the line reads safe_automation@5%=none. check --min-safe-rate 0.10:0.60 exits 0 here and exits 1 if that share falls below 60 %. To audit a real vendor, see docs/real-audits.md.

In the GitHub Action the same controls are inputs: target (the error rate the safe automation rate is headlined at, default 0.05), segment-by (where to look for the worst segment: label, meta.FIELD or none) and min-safe-rate (RISK:SHARE minimums that fail the job). It returns safe-automation-rate (at the target) and safe-automation-rates (every target) as outputs.

What you get

The HTML report: key figures (safe automation rate, accuracy, calibration error, cost and latency), data and ground truth, then the answer to Can I automate this?

One self-contained report per run (Markdown or HTML, prints to A4): the answer first: the share validated on held-out decisions, with its approximate bound and its range over split seeds, then the threshold to deploy; calibration with intervals; accuracy against coverage; cost and latency; the provenance of every number. Shown here on the seeded simulator, stamped SIMULATED. audit-result.json carries the same numbers for your pipeline.

How it works

  1. Labeled dataset — JSONL rows {state, questions, labels}. Your humans already decided; the judge must reproduce them. An optional first line declares where the labels come from — their ground-truth tier, from GT-1 constructed to GT-6 production outcome, and what each tier lets you claim.
  2. Shadow run — the judge scores every row. Nothing is automated; everything is recorded: decision, confidence, latency, cost, raw response.
  3. Audit report — calibration (ECE with equal-width and with equal-mass bins, Brier score, reliability bins — three numbers, plus log loss next to Brier, never combined), selective prediction (accuracy at every coverage level), cost, latency percentiles, plus provenance: model, backend, timestamp, dataset hash. Accuracy, the three calibration numbers and zero-error coverage carry a 95 % bootstrap interval over the dataset's distinct texts (Jev on the bare-label router: 66.7% [52.5, 80.3]; 2,000 seeded resamples, pure Python, --no-ci to skip), or the exact binomial interval where the bootstrap cannot move so a number on 120 rows reads as the noisy estimate it is. Every report header carries a Ground truth: GT-1 constructed — … line with the tier's meaning and the dataset's caveats, so two accuracies are never read as the same evidence; a dataset that declares nothing is reported as GT-0 unknown, never silently.
  4. CI gate — judge-audit check --baseline baseline.json --max-ece-drift 0.02 fails the build when the judge degrades.

Exit codes: 0 ok · 1 drift detected · 2 usage or configuration error, or a judge that could not answer (an HTTP error, an expired key, a server that cannot be reached; one line, judge request failed: …, no traceback). The message says what to fix. An output that cannot be written exits 2 even when drift was also found: fix it before reading the drift.

In CI: the GitHub Action runs the audit on every push or pull request and fails the build on drift:

- uses: kunko-ai-labs/judge-audit@v0.6      # or pin the release's commit SHA
  with: { labels: audits/labels.jsonl, judge: jev, baseline: audits/baseline.json,
          target: "0.05", segment-by: label, min-safe-rate: "0.05:0.40" }
  env: { AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} }

One sticky PR comment that opens with the safe automation rate validated on held-out decisions (approximate), then the audit table (n, accuracy, ECE, cost, p99, verdict vs baseline) and, labelled as observed on all rows without a bound, the zero-error coverage; the report in the job summary, the report and result JSON as an artifact. The per-decision judgments stay on the runner unless you ask for them (upload-evidence: "true"). min-safe-rate: "0.05:0.40" fails the job unless at least 40 % of the labelled decisions is automated at a 5 % error target (inputs in the Quickstart). Outputs accuracy, ece, zero-error-coverage, safe-automation-rate (at the target), safe-automation-rates (every target), drift for anything downstream. Details in docs/integrations.md.

From inside an agent: pip install "kunko-judge-audit[mcp]" then claude mcp add judge-audit -- judge-audit-mcp (or the equivalent in Cursor). The agent gets run_audit, check_drift and list_judges and can audit the judge it is about to rely on without leaving the session. See docs/integrations.md.

Judge interface

Anything that maps (state, questions) -> (decision, confidence) plugs in:

from judge_audit import Judge, Question, Judgment

class MyJudge(Judge):
    name = "my-judge"

    def decide(self, state: str, questions: list[Question]) -> list[Judgment]:
        ...  # return Judgment(question=q.name, decision="spam", confidence=0.93)

Ships with fourteen: jev (TypeSafe Jev — and, via JEV_ENDPOINT, any Jev-compatible server such as OpenJev), llm (any chat model with a confidence prompt: Claude through the official SDK, anything OpenAI-compatible — OpenAI, Gemini, Ollama, vLLM — or your own transport), nli (a local zero-shot encoder, the control), finetuned (your own classifier, trained on your labels with scripts/train_classifier.py), laya (Laya, an open-weight judgment model run locally), decision2 (Decision 2.0, an open decision model run locally at a pinned revision), decider (decider, likewise), strands (Strands Decider, likewise), clef (Clef and Clef-flash, likewise), openai-decisions (the OpenAI Decisions API, hosted), clef-hosted (Clef and Clef-flash on Workers AI, hosted), microsoft-decision (Microsoft-Decision-1 through OpenRouter, hosted), logprob (an open model's own probability of each option, run locally with MLX) and simulated. Details in docs/judges.md.

Learn more

Results by version v0.5: pre-registered study on real data · v0.4: emails under attack, Jev audits, the Arena, the consensus panel
How the metrics work docs/judges.md: calibration, the safe automation rate, intervals, what each number can and cannot say
Integrations GitHub Action, MCP server, Python API · auditing a real vendor
Roadmap and changes ROADMAP · CHANGELOG · releasing and supply chain
Contributing CONTRIBUTING: what helps most and how review works

Why calibration, not accuracy? Accuracy tells you who wins a benchmark. Calibration tells you what you can automate safely. A judge that is confident on the decisions it gets wrong automates its own mistakes; one that flags its own doubt hands those to people. Audit the honesty, not the score.

FAQ

Is this a benchmark? No. A benchmark ranks judges on a fixed test. An audit checks one judge on your decisions and tells you what you can automate. The datasets here are worked examples, not a leaderboard — yet.

Why was Jev audited first? Its vendor sells calibrated confidence as the headline feature. A claim that specific deserves an independent check.

Can I trust a 100 % result? Only as far as the dataset. The clean-email audit says the judge handles templated business email; it says nothing about your inbox. That is why the adversarial and routing audits exist.

Does it send my data anywhere? Only to the judge endpoint you configure. Reports and checkpoints are local files; commit them or not. In CI the Action uploads the report and the metrics JSON as a run artifact; the per-decision judgments are uploaded only if you set upload-evidence: "true".

License

Apache-2.0 — see LICENSE. Sister project: agent-assurance (deterministic checks on what an agent may do; this repo is the probabilistic half: whether its judgment can be trusted).

Metadata

Release files for kunko-judge-audit 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kunko-judge-audit 0.6.1
File Size Uploaded Provenance Actions
kunko_judge_audit-0.6.1.tar.gz 205.1 kB With provenance Details for kunko_judge_audit-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kunko-judge-audit 0.6.1
File Interpreter ABI Platform Actions
kunko_judge_audit-0.6.1-py3-none-any.whl Python 3 none any Details for kunko_judge_audit-0.6.1-py3-none-any.whl

Total release size: 376.1 kB

Release files / kunko_judge_audit-0.6.1.tar.gz

Download URL kunko_judge_audit-0.6.1.tar.gz
Size 205.1 kB
Tags Source
SHA-256 checksum
How to use checksums
fd4066a81fa13d33c1fa13df590d37abaaac81340c77293f485454a3796cfb18
BLAKE2b-256 checksum
How to use checksums
9b16e4d568a94e504e2c63d85f9ed627bf5b53a351994c1d3b87f7fc899a95c6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Release files / kunko_judge_audit-0.6.1-py3-none-any.whl

Download URL kunko_judge_audit-0.6.1-py3-none-any.whl
Size 171.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
37da6bb68379a429c676fbea822257ed0196769978907fdecfede2c8a2caa5b6
BLAKE2b-256 checksum
How to use checksums
f8eb0a8c75d92ccf92965fb0947eee278d4b4b57d69946a823ca8f166a553034
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Security for kunko-judge-audit 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for kunko_judge_audit-0.6.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page