judgeman

Step-level evals for AI coding agents, with a measured number for how far to trust the judge.

Most agent evals score the final result. judgeman reads every step of a run, flags the ones that were wasted, repeated, dangerous or submitted unverified, and tells you how often its verdicts match a person's, on public runs and, after a short calibration, on your own agent.

Works on logs you already have: Claude Code sessions, OpenAI Agents SDK sessions, OpenTelemetry traces (OpenInference or GenAI conventions), mini-SWE-agent trajectories.

Install

pip install 'judgeman[judge]'      # or: uv tool install 'judgeman[judge]'
export OPENROUTER_API_KEY=...      # any OpenAI-compatible endpoint works, see below

Five minutes on your own agent

# 1. Judge a folder of runs. Rules on every step (free), a 9B open judge on every step (cents),
#    a frontier judge only on the steps either of them flagged (about one in three). Capped at $1.
judgeman judge ~/.claude/projects/-Users-you-code-myproject --out judged.jsonl

# 2. Read the verdicts: one self-contained HTML page, or the terminal.
judgeman report ~/.claude/projects/-Users-you-code-myproject --labels judged.jsonl --out report.html
judgeman show   ~/.claude/projects/-Users-you-code-myproject --labels judged.jsonl --flagged

# 3. Find out how far to trust it on *your* agent: label 50 blind steps (20 minutes),
#    get agreement per axis and per layer, with one line of advice each.
judgeman calibrate ~/.claude/projects/-Users-you-code-myproject --judged judged.jsonl

# 4. Gate CI. Exit 1 on any dangerous or unverified step; opt in to a wasted-step budget.
judgeman gate runs/ --labels judged.jsonl --max-wasted 5

judgeman judge --dry-run counts steps and tokens without spending anything. For a local model set JUDGEMAN_BASE_URL=http://localhost:11434/v1 and pass --small <ollama model>.

Where the logs are

A step is one tool call and its result. Non-shell tools become pseudo-commands the rules understand (Read path, Edit path, Grep pattern), so the same checks run on every agent. Pass or fail per run comes from a <run>.result.json beside the log ({"resolved": true}) or a per_instance_details.json; without it the outcome axis is skipped. examples/agents-sdk/ holds four recorded runs in two formats to try the commands on.

What a step is judged on

All yes/no. A missing value means the axis does not apply to that step.

The free layer: rules

The checks are tuned to rarely fire wrongly, even if they miss things. When a frontier judge is asked about a flagged step, its verdict replaces the rule's (in the example runs it clears an edited_existing_test on a conftest the agent had to create).

How far to trust it

Measured on 516 hand-labeled steps from published mini-SWE-agent runs on SWE-bench Verified (six models), plus 196 runs with planted mistakes. Full method and tables in reports/leg1-report.pdf and reports/leg2-report.pdf.

Read the layered row with care: it was measured on the same labels the layering rule was chosen from, and its interval overlaps the frontier-only row. The honest number for your agent is the one judgeman calibrate prints.

On a different agent. judgeman calibrate was run on the Claude Code session that built this repository (380 steps, Claude Fable 5.1), with 50 blind steps labeled by hand, 35 minutes and $3.55. On risky the layered verdict reached kappa 0.70 against those labels: it caught all 10 steps the labeler called dangerous (scripts that rewrote the test file, flagged by edited_existing_test and confirmed by the frontier judge) and raised 8 the labeler did not mind (pushes to main, rm -rf dist, a git checkout -- that discarded edits). On progress the labeler found no wasted step in the sample while the judge called 8 of them wasted: empty tool-loading calls, commands the harness blocked, a script that crashed. Too few to compute a kappa, and the tool said so. The direction was clear enough to act on: the Claude Code adapter now skips tool-loading calls, the one kind of harness no-op the agent never chose.

What did not help: showing the judge what the agent did next (hindsight) lowered agreement; mid-tier models were no better than the cheap one; two QLoRA fine-tunes of the 9B judge on 295 to 400 labels did not beat the base model on wasted steps (kappa 0.22 and 0.18 vs 0.20), because both learned the one thing labelers disagree on, that a failed command is fine. The fine-tuned judge is still useful next to the rules: it caught 97 to 100% of planted destructive commands it had never seen, with 4 false alarms in 513 clean steps. Adapter and card: Christopher4113/judgeman-qwen3.5-9b-judge.

Known gaps

• risky is unmeasured on real runs: 516 sandboxed SWE-bench steps hold no dangerous step. The numbers above for it come from planted mistakes only.
• redundant is not yet a usable axis. Labeled with the run history on screen, the labeler marks about a third of steps as repeats; Opus 5.5 reaches kappa 0.20 against that, the cheap judge 0.02 and the checks 0.12. The definition needs tightening before any judge is compared on it.
• Failed commands: two labelers agree on whether a failure was wasted only at kappa 0.39, so judgeman reports them as their own kind (failed_command) and does not escalate on them.
• judgeman agree marks an axis with * when it has fewer than 10 yes or 10 no labels. Treat those numbers as unknown.

Every command

Judge calls are cached in .judgeman/cache/ keyed on model, prompt and step, so re-runs are free.

Develop

uv sync --extra judge
uv run ruff check . && uv run pytest -q

DEMO.md is the three-minute walkthrough. Labels, queues and every judge's verdicts from the study are in labels/. Apache-2.0.