eval-builder

Your agent turns your app's real logs into an eval suite, and tells you which of its judges you can actually trust.

Try it on 48 sample conversations (no account, no API key, no model needed for these steps):

curl -sLO https://raw.githubusercontent.com/Abelo9996/eval-builder/main/examples/sample_logs.jsonl
uvx eval-builder ingest sample_logs.jsonl     # or your own logs: OpenAI, Anthropic, Langfuse, OpenTelemetry, JSONL
uvx eval-builder select -n 10 --stratify category && uvx eval-builder draft

What you get (real output, eval-builder 0.1.1 on the sample file):

ingested 48 traces into evalset/traces.jsonl
  sample_logs.jsonl: format=openai records=48 traces=48 skipped=0 sha256=2d5dd2295784988b
redactions: 0 {}
48 traces -> 16 unique (32 exact dupes, 0 near dupes) -> selected 10 (8 failures) across 4 clusters
  q121-llama-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
  q102-alpaca-13b: covers category=reasoning (2 unique traces, 12%)
  q111-alpaca-13b: adds variety within cluster 1 (8 traces, 50%; triangle, response, person); least similar to cases already picked there
  ...
10 case(s) added, 10 total in evalset/cases.yaml

evalset/cases.yaml now holds 10 real conversations with a TODO where the expected behavior goes. Writing that, running judges and reading the statistics is the part your coding agent does, through the MCP server (next section).

eval-builder is not another eval platform. It builds the suite and checks the judges, then exports to the tools you already run: promptfoo, DeepEval, Inspect AI, or plain JSONL. It never calls a model. Your coding agent (Claude Code, Codex, Cursor) does the thinking; eval-builder does the selection, the bookkeeping and the statistics, and writes down the evidence.

Use it with your agent

uvx eval-builder setup --yes    # registers the MCP server and the skill, then restart your agent

Then type something like this into Claude Code, from the folder that has your logs:

Our assistant's chat logs from last month are in ./logs. Build a small eval suite from them (about 10 cases) that I can run in promptfoo, and tell me whether I can trust a local judge for grading it. I have Ollama running with qwen2.5:7b-instruct and llama3.2:3b. Keep the judge runs small.

This is from a real headless Claude Code session (October 2026, Claude Code 2.1.293) given exactly that prompt and 48 conversations from examples/sample_logs.jsonl. It ran ingest, select, draft, list_cases, set_rubric, ten update_case calls, validate, judge_plan, 140 judge calls to Ollama through a short script it wrote, judge_check, export and report in 32 turns and about 10 minutes. From its final answer:

I built a 10-case promptfoo suite, but you can't trust either local judge yet. Neither one has been checked against human labels, and each confidently got an obvious case wrong.

judge	flip rate	padding helped	verdict
qwen2.5:7b-instruct	10% [2%, 40%] (n=10)	0% [0%, 28%]	not_enough_data
llama3.2:3b	20% [6%, 51%] (n=10)	0% [0%, 28%]	not_enough_data

qwen7b failed a clear case. It passed the Hawaii rewrite 5 out of 5 times, though only the first sentence starts with "A".

No grader is wired in. Because no judge passed, the llm-rubric checks fall back to promptfoo's default grader, which needs an OpenAI key. To grade locally anyway, export again with the judge set explicitly (the export tool's judge option, e.g. qwen7b). Its verdicts will be unverified.

It wrote every expected behavior itself because nobody was there to confirm them, and said so on each case. It did not invent human labels; it asked for them.

An earlier session against 0.1.0 is where the promptfoo context bug fixed in 0.1.1 came from. The agent read the export and told the user: "Right now it only sends the final question ({{input}}), so your app won't see what "it" refers to in "Can you parallelize it?"."

Example: real output

Everything below comes from runs on this machine (Apple M4, 16 GB) that are committed in examples/mt-bench/. The data is the LMSYS MT-Bench human judgments dataset (CC BY 4.0): 80 two-turn questions answered by 6 models, plus pairwise verdicts from expert human judges.

Logs to suite. ingest read 480 conversations (OpenAI chat format, redaction on, 0 matches). select -n 24 --stratify category removed 400 exact duplicates (each question was asked to six models), clustered the 80 unique conversations into 9 topics and picked 24 covering all 8 categories, 19 of them failures. From suite/report.md:

| trace           | cluster | represents | why it was picked |
| q81-alpaca-13b  | 0 | 6 | failure (negative user feedback); 62% of unique traces are failures and at least 30% of picks are reserved for them |
| q82-alpaca-13b  | 0 | 6 | adds variety within cluster 0 (11 traces, 14%; response, previous response, previous); least similar to cases already picked there |
| q156-alpaca-13b | 3 | 6 | covers category=humanities (10 unique traces, 12%) |

The agent wrote expected behavior for the 24 cases, and export produced files that Inspect AI 0.3.277, DeepEval 4.2.8 and promptfoo 0.124.0 all load (see verify_exports.sh).

Which judges can you trust? Five local LLM judge setups (four models through Ollama, one of them also at temperature 0) and two controls that are not language models judged 80 answer pairs where human experts picked a winner. Each judge ran 5 times per pair, 5 more with the answers swapped, and 5 more with an irrelevant paragraph appended to one answer: 8,400 calls, 0 errors. eval-builder judge-check, from judges/report.md:

n = 80 pairs per judge; brackets are 95% intervals. No judge passed. The closest, qwen2.5 7B, reaches kappa 0.40 to 0.45 with the human experts, but its verdict flips on a quarter of the pairs when the answer order is swapped, and setting the temperature to 0 removes the run-to-run flips without removing that order effect. llama3.2 3B picked whichever answer it saw second 80% of the time. The longer-answer rule never flips and agrees with humans 66% of the time, which is why stability and accuracy alone are not enough: the padding probe catches it.

The same check on the suite's own pass/fail judge (qwen2.5 7B, 24 cases, no human labels) gives unstable: its verdict changed across 5 identical calls on 7 of 24 cases (29%, interval 15% to 49%).

How it works

The verdict thresholds are explicit flags with defaults: flip rate at most 20% of cases, position consistency at least 80%, padding helps at most 10% of cases, kappa at least 0.4 on at least 20 human-labeled cases. A judge without human labels is never called trustworthy.

Setup for agents

uvx eval-builder setup          # shows what it would change
uvx eval-builder setup --yes    # applies it; restart your agent afterwards

setup registers the MCP server (uvx eval-builder mcp) with Claude Code (claude mcp add --scope user), Codex ([mcp_servers.eval-builder] in ~/.codex/config.toml) and Cursor (~/.cursor/mcp.json), and copies the agent instructions to ~/.claude/skills/eval-builder/ and ~/.codex/skills/eval-builder/. It backs up any file it edits and does nothing on a second run. The workflow the agent follows is in skills/eval-builder/SKILL.md.

MCP tools: ingest, select, draft, list_cases, update_case, set_rubric, validate, judge_plan, judge_check, export, report, status. The judge runner is CLI only, because it executes a command.

What it can't do

• It does not write expected behavior or human labels. The agent drafts expected behavior with you; labels must come from people. Without labels, judge-check can tell you a judge is unstable or biased, but not that it is right.
• Selection is lexical. Two requests that mean the same thing in different words can land in different clusters, and near-duplicate detection only catches close textual matches.
• Redaction is pattern-based. Names, street addresses and free-form secrets get through. Look at traces.jsonl before sharing a workspace.
• The bias probes cover answer order and irrelevant length only. Self-preference, style bias and rubric misreadings are not measured.
• Small samples give wide intervals. The report prints them; read them.
• It does not run your app, your judges or your eval. The agent (or a script you name with judge-run) calls the judge model; the exported files run the eval in promptfoo, DeepEval or Inspect AI.
• Only the promptfoo export wires a checked judge in as the grader, and only a pointwise judge whose prompt answers in JSON ({"pass": ..., "reason": ...}), because promptfoo's llm-rubric cannot parse a bare "pass". DeepEval and Inspect exports use their own default graders.

Privacy and safety

Everything runs locally. eval-builder makes no network calls and sends nothing anywhere. The only process it starts is the judge command you name with judge-run --enable-judge-plugin. Redaction is on unless you pass --no-redact, and the report lists what was replaced (counts and kinds, never the values).

License

MIT. See LICENSE.