Five-minute demo  ·  The checks  ·  The judge's exam  ·  Newsletter

No API keys, ever, by default. No platform. One markdown report.

A performance review for your AI agent: the checks that matter before it touches production. One CLI, one YAML test file, one markdown report.

Point it at your RAG bot's HTTP endpoint with a hand-written YAML test file, and it writes report.md: a one-page verdict a stakeholder can read without installing anything, with the failing answers quoted verbatim and a line per failure saying whether retrieval or generation broke. The report commits to git and diffs in pull requests. The judge is a local Ollama model, and the tool grades its own judge before it grades you.

agent-report-card run --tests board_questions.yaml --endpoint http://localhost:8000
→ accuracy 84% (16/19) · hallucination 6% (1/16) · 3 failures · NOT READY → report.md

Those numbers are real output from the bundled demo (a fixture bot with deliberately planted flaws; see below). CI re-runs the deterministic numbers (the accuracy and the failure count) on every push; the judged 6% is pinned to the committed judged report in reports/, which CI checks this README against.

🧭 Why this exists

Every eval tool hands you floats, a DataFrame, a web grid, or a SaaS share link. Nothing hands a non-engineer a one-page markdown verdict with the failures quoted, which is the document teams hand-write before every "is it safe to ship" meeting. This tool treats that document as the product. And with hosted eval platforms being acquired and shut down, your quality bar should not be someone else's product: this is MIT, local, and its output is a text file that cannot be deprecated.

⚡ Five minutes, no model, no keys

pip install agent-report-card
agent-report-card demo --judge none

Python 3.10 or newer (stock macOS python3 may be 3.9; use a python3.11, pyenv, or uv interpreter in that case). To work on the tool itself, clone it and pip install -e . instead.

Then take the twenty-minute tour: examples/ walks from a four-question suite to a CI gate, one runnable file at a time, and explains where the test cases themselves come from, which is the part no tool does for you. Same tour as an executable notebook in examples/tutorial.ipynb, with every output produced by a real run.

That boots a bundled fixture bot (a fake support bot for the fictional Northstar Telecom, with nine flaws planted on purpose), runs the bundled 21-question suite against it, and writes report.md. Fully offline, and every report prints its own grading run in the header, which is how long grading took rather than how fast the bot answered (the committed deterministic run measured under a second, plus interpreter startup; the bot's own latency is a row in the scorecard). With Ollama running, drop the --judge none and the judge checks light up too; the committed judged run's figure is in its own header, because judged runs on a 27B local model are minutes, not seconds, and pretending otherwise would break house rules.

To try the promised command exactly as written above:

agent-report-card demo --keep-serving --port 8000
# in another terminal, from the same directory:
agent-report-card run --tests board_questions.yaml --endpoint http://localhost:8000 --judge none

(--port 8000 errors clearly if something else owns the port; without the flag, demo falls back to a free port and tells you. Drop --judge none when Ollama is up; without Ollama the run exits 2 with a message naming the fallback, because run never silently downgrades.)

See a report before installing anything — one committed example per verdict, all real runs against the bundled bot:

Each log is the exact command and per-case console stream that produced its report, ending with the exit code (0 for both PASS levels, 1 for NOT READY: what CI keys on). scripts/make_gallery.py regenerates them.

Pass --html report.html and you also get the same report as one self-contained HTML file for the stakeholder who would rather receive an attachment than a markdown file: inline CSS, zero JavaScript, no external requests, and a print stylesheet so it becomes a clean PDF. GitHub shows .html as source, so download it or open it locally to see it rendered. It is a second rendering of the same document, never a different one, and a test asserts the two carry identical numbers.

🔌 What your endpoint must return

The tool speaks HTTP: one POST per question, JSON in and out. The default contract is:

POST {endpoint}/ask          {"question": "..."}
  -> {"answer": "...",
      "contexts": [{"text": "...", "source": "doc.md"}, ...]}   # optional

contexts unlocks the grounding and retrieval-localization checks; source fields unlock the citation checks. Without contexts, the checks that need them render as n/a with the reason, never as passes. Sources are the exception: cites_expected_source fails, rather than rendering n/a, on any case that lists expected_sources, because citing nothing is a real answer to "did it cite the right document". The mappings above are the code's actual defaults (set contexts: null in the YAML to disable one), and different shapes map via dot-paths (answer: choices.0.message.content covers any OpenAI-compatible server), so the system under test can be written in any language. agent-report-card init writes a commented starter YAML showing every supported field.

Test files are strictly validated: unknown keys, duplicate ids, and contradictory fields fail fast with the line number, the case id, and a one-line fix, before a single request is sent. And because JSON is a subset of YAML, --tests suite.json works through the same loader with the same validation.

📋 The checks that matter

14 deterministic checks (pure Python) and 5 judge checks (local LLM, binary verdicts only), fixed by design: no plugin API, no 50-metric buffet. The full catalog with what each check catches is in CHECKS.md, generated from the same table the code runs. Accuracy is always computed by both routes, deterministic and judge, and shown side by side; when they disagree, the case lands in a "needs human review" section instead of being averaged away. The deterministic route wins the verdict, and the judge is never the sole authority on a number.

The verdict's hard gates come from your YAML gates block, or from this tool's stated defaults when you omit it (accuracy at least 80%, hallucination at most 5%, critical cases must pass), and every report's "Bars used" line says which of the two it applied. So the exit code gates CI honestly: 0 pass (with or without warnings), 1 gate failed, 2 tool error. One exception, which the report states on the gate line itself: when the judge has failed its own exam, a breached hallucination ceiling downgrades to a warning rather than a gate failure, so the exit code stays 0 and the quoted cases need a human. Warnings (route disagreements, judge errors, an unevaluable judge-fed gate) soften PASS to PASS WITH WARNINGS without touching the exit code, and the banner's failure count is defined as answerable cases whose correctness verdict failed; the report spells that out next to the full quoted-failure list.

⚖ The judge takes the exam it grades

Every report names its judge, its temperature, and its prompt hash, and embeds the judge's score on a 30-pair hand-labeled exam (agent-report-card judge-check), including five answers that are wrong by less than 1%, the kind of error judges miss most. On this machine, qwen3.6:27b scored 30/30 (zero false passes, zero false fails, 5/5 on the subtle numerics); the committed result is in reports/. Thirty items supports "measured error rate on this labeled set", not "calibrated", and below 80% agreement the verdict line itself tells you to trust the deterministic column.

One honest war story from building this: the first exam run used the judge's default thinking mode, and 12 of 30 calls blew the per-call timeout (reports/calibration_first_run_thinking_enabled.log, the captured output: agreement 18/30, 4824s). The tool recorded every timeout as judge_error rather than guessing, exactly as designed, and the fix (judge calls now ask Ollama to skip thinking, with an automatic fallback for models that reject the field) took the same exam to 233.6s at 30/30, per the committed calibration JSON. The judge's own report card caught the judge's own failure before it graded anything real.

🌏 Languages

Everything is UTF-8. The deterministic checks are substring- and number-based with NFC normalization, which makes them well-defined for unsegmented Thai text (tested with Thai fixtures; note a substring can match across Thai word boundaries). Refusal detection extends to any language via the patterns block. The judge handles whatever languages your local model handles; the calibration exam is English-only in v0.1 and every report says so.

🚫 What this deliberately is not

No dashboard, no web UI, no cloud sync, no comparison matrices, no synthetic test generation, no plugin API, no security scanning. Those exist elsewhere and most of them are the reason this tool exists. CI on the free GitHub runner covers everything except live-judge calls (no Ollama there); judged artifacts are produced locally and committed, and scripts/verify_readme_claims.py fails CI when the load-bearing numbers this README quotes (the banner's accuracy and failure count, the judged hallucination fraction, the calibration score and subtle-numeric recall) drift from a fresh run or from the committed artifact behind them.

🗺️ Roadmap

Today (v0.1.2) you can grade a RAG endpoint against a YAML suite and get a markdown or HTML verdict, with a local judge that reports its own error rate. Per-release detail lives in CHANGELOG.md; this section is only about what comes next.

v0.2.0 · the judge's exam, reported properly. Every report already prints how the judge scored on 30 hand-labeled pairs. One number hides the failure that matters: a judge that gets 28 of 30 right while missing four of the five subtle-numeric pairs reads as 93% correct and is useless at exactly the job this tool exists for. So the exam result breaks down by category, agreement is corrected for chance (a coin flip scores about 50% on a balanced set), every rate carries its denominator and an interval, and no percentage is printed for a category with two examples in it. A comparison between two judges says "too close to separate on 30 items" rather than ranking them, because 30 pairs can show a judge is usable and cannot show one is two points better than another.

v0.2.x · any judge you want to run. Pluggable judge providers: OpenAI-compatible servers so LM Studio and vLLM work without a wrapper, and hosted APIs for anyone who would rather not run a 27B model locally. Your key, your environment, read from one variable you set and never from a flag. Local stays the default, always. This is catch-up work rather than a reason to switch: other tools already judge with whatever model you like. What stays different here is which way the default points, and that the report tells you which judge graded the run and whether anything left your machine.

v0.3 · regression gating. Right now a run tells you where you stand, not whether you slipped. This diffs a run against a stored baseline, so CI can fail on "worse than last week" rather than only on an absolute floor. The scores sidecar already carries everything needed; it is marked unstable precisely so this can shape it. Wanted for sovereign-rag, which needs a regression gate rather than a threshold.

When it is unblocked · agent-trace mode. Today the tool grades one question and one answer. A multi-step agent fails differently: step four goes wrong and steps five through ten inherit it, so a single end-to-end score tells you nothing about where to look. This ingests a run log and scores per step, surfacing the compound-failure profile that agent-failure-lab measures. Blocked on that repo versioning its JSONL format, because building against an unversioned schema would just move the breakage here.

Small things, if someone asks. A completeness check for multi-part questions, and grading a bare model endpoint directly. Nobody is blocked on either, so they wait for a real request. The second one comes with a warning: a bare model returns no retrieved context, so most of the checks would render n/a and the report would grade generation only.

Deliberately not coming, so you can plan around it: no dashboard, no hosted service, no plugin API, no model-versus-prompt comparison matrices, no synthetic test generation, no red-teaming. Those exist elsewhere and are mostly why this tool is small. If the report ever needs a server to read it, this project has failed.

Written by Satsawat Natakarnkitkul, author of Why Your AI Agent Will Fail. Companion article: "Why Your AI Agent Needs a Performance Review (Literally)" at satsawat.ai · Newsletter: AI in Practice

License: MIT