eval-doctor 0.1.2

Know what you are not testing before you ship - an eval coverage auditor for RAG and agent systems.

Eval Doctor

Know what you are not testing before you ship.

Eval Doctor is a CLI that helps AI builders find blind spots in their eval suite — a linter for AI evals.

Most AI projects start with architecture and implementation. Evals come later, once the system already "kind of works." But for RAG systems and agents, the hardest part is not writing test cases — it's knowing what failure modes matter. Eval Doctor scans your repo, detects your AI system's capabilities, maps them to expected evaluation dimensions, and tells you what your current eval suite is missing.

You are testing faithfulness and answer relevance.

You are not testing:
- empty retrieval
- stale documents
- tool timeout
- hallucinated tool arguments
- prompt injection
- ambiguous user requests

Install

pip install eval-doctor               # static analysis, no API keys needed
pip install "eval-doctor[anthropic]"  # + optional LLM-powered capability detection (Claude API)
pip install "eval-doctor[openai]"     # + optional LLM-powered capability detection (OpenAI API)

Already have Claude Code installed? No extra or API key needed — eval-doctor audit --llm uses your Claude subscription through the claude CLI.

Usage

cd your-ai-project
eval-doctor audit

You get a console report plus two files:

• eval_doctor_report.md — human-readable coverage report
• eval_doctor_findings.json — machine-readable findings for automation

Options:

eval-doctor audit --repo path/to/repo      # audit a different directory
eval-doctor audit --evals path/to/evals    # eval files living outside the repo
eval-doctor audit --output-dir reports/    # where to write report files
eval-doctor audit --llm                    # force LLM enrichment (also enables the claude CLI path)
eval-doctor audit --no-llm                 # force pure static analysis
eval-doctor audit --force                  # overwrite existing report files
eval-doctor interview                      # short risk interview; audit picks up the answers
eval-doctor show-taxonomy                  # print the built-in taxonomy

Risk interview

Severity defaults are generic; your risks are not. eval-doctor interview asks a few multiple-choice questions keyed to what the scanner actually found in your repo — retrieval, web grounding, tool calling, SQL, memory — plus where the system will run, and writes eval_doctor_risk.json. The next eval-doctor audit picks it up automatically (or point at one with --risk path/to/eval_doctor_risk.json) and elevates the severity of the dimensions you said would hurt most, re-ordering the report. Answers only ever raise severity — the interview never argues a risk away.

How it works

Repo Scanner ──► Capability Extractor ──► Coverage Engine ──► Report
(deterministic)  (static rules,           (expected vs
                  optional LLM enrich)     observed dimensions)

1. Scan — a deterministic scanner walks your repo and detects signals: frameworks (LangChain, LlamaIndex, CrewAI, ...), vector DBs, embeddings, chunking, reranking, tool definitions, SQL, browser automation, memory, prompt templates, and existing eval/test files. Nothing leaves your machine.
2. Extract — signals are mapped to a capability profile (rag, tool_calling_agent, sql_agent, ...).
3. Compare — the profile selects applicable dimensions from a built-in taxonomy of 40 eval dimensions across RAG, tool calling, SQL agents, memory, and general AI risks. Your existing eval files are checked for evidence of each.
4. Report — every applicable dimension is rated covered, weak, or missing, with a severity and a concrete recommendation.

Optional LLM enhancement

Static analysis sees imports and patterns, not intent. With an LLM available, Eval Doctor sends its collected evidence snippets (never your whole repo) to a model to correct and complete the capability profile — catching things like web-search-grounded generation that deserves faithfulness evals, hand-rolled RAG with no framework, and pruning false positives like an sqlite cache being flagged as a SQL agent. Two ways to enable it:

• API key — set ANTHROPIC_API_KEY or OPENAI_API_KEY and install the matching extra. Used automatically when present.
• Claude subscription — have Claude Code installed and signed in, then pass --llm. Eval Doctor shells out to claude on your machine; no API key or extra needed. This path is opt-in only — an installed CLI is never used unless you ask.

Without either, everything still works; the report just notes it ran in static-only mode.

Additional providers are welcome contributions: implement the two-method LLMProvider protocol in eval_doctor/llm.py — the existing providers are small, self-contained classes.

Example

See examples/sample_report.md for the report produced by auditing a small RAG + tool-calling fixture project.

Roadmap

• Candidate eval generation — generate golden eval cases for your missing dimensions
• Coverage diff — eval-doctor diff to score an eval suite against recommended coverage
• CI / PR mode — comment on PRs that add AI capabilities without corresponding evals

Development

pip install -e ".[dev]"
pytest

License

MIT