Agent Evaluator

Harness Engineering evaluation SDK that judges AI agent deployment readiness through 7 Gates.

It asks not just "does the agent work well?" but "is the agent ready for production?" One decorator line auto-recognizes 24 frameworks (LangChain, CrewAI, AutoGen, …) and measures 58 metrics (25 Native Trackers + 33 Harness Config) without touching your agent code — then aggregates them into 7 Gate pass/warn/fail judgments, a root-cause diagnosis engine for regressions, and statistically valid A/B testing.

pip install agent-evaluator

from agent_evaluator import QuickEval

eval = QuickEval("results/")

@eval.qa
def my_agent(question: str, ground_truth: str = "") -> str:
    return llm.invoke(question)          # your agent code — unchanged

my_agent("What is the capital of South Korea?", ground_truth="Seoul")

eval.save()                                        # results/quickeval.json + .html
eval.gate(tcr=85, accuracy=70, hallucination=5)    # CI/CD gate — sys.exit(1) if unmet

The 7 Harness Gates

Pass any of the 33 Configs above as @agent_eval/@batch_eval/@conversation_eval parameters and PerformanceMonitor auto-aggregates each Gate's pass/warn/fail from the underlying trackers — no separate scoring pass needed.

@agent_eval(monitor, task_type="qa",
    instructions=InstructionConfig(required_keywords=["Seoul"], fail_on_violation=True),   # Gate A
    loop_detection=LoopDetectionConfig(consecutive_repeat_threshold=6),                    # Gate B
    sla=SLAConfig(p95_ms=3000),                                                            # Gate D
)
def my_agent(question: str, ground_truth: str = "") -> str: ...

Full Gate reference: Docs/05_QUALITY_GATE.md · Runnable walkthrough: Evaluator_Examples/ch03_harness_basics.py

What's Inside

• 3 decorator types — @agent_eval (1 call → 1 result), @batch_eval (1 call → N results), @conversation_eval (N calls → 1 multi-turn result). All non-invasive: your function's signature, return value, and exceptions are untouched. → Docs/01_GETTING_STARTED.md
• 24 framework adapters — framework="langchain"/"crewai"/"anthropic"/"openai"/… auto-extracts tool_calls/chain_steps/tokens_used from the framework's native response object (duck typing — works without agent-evaluator importing the framework itself). → Docs/03_INTEGRATION_GUIDE.md
• 58 metrics — 25 Native Trackers (accuracy, hallucination, latency, tool efficiency, 5 security trackers, …) + the 33 Harness Configs above. → Docs/02_METRICS_GUIDE.md or the in-app SDK Reference (agent-eval dashboard → /sdk-docs)
• Self-contained HTML report — save_to_file() (and agent-eval gate) write one .html next to the JSON — no server needed. It leads with a one-line deployment-readiness verdict + a HIGH/MEDIUM/LOW confidence badge, then Next actions 1·2·3, a Path to Green (quantified gap to each failing gate + an impact-ordered fix plan), per-gate Score Breakdown, the worst failure cases each with a tool-call trajectory waterfall and accuracy-signal breakdown, and Recommendations carrying paste-ready @agent_eval snippets. Pass a baseline and it adds the regressed/new/fixed failure-set diff plus prompt/config change attribution. → Docs/09_OUTPUTS.md
• CI/CD quality gating — agent-eval gate result.json --tcr 85 --accuracy 70, plus baseline regression detection, per-version baselines, and golden-set regression gating. → Docs/05_QUALITY_GATE.md
• Root-cause diagnosis (RCA) — agent-eval diagnose / agent_evaluator.rca.diagnose() automates detect → attribute → cross-reference for a Gate regression, and links Gate F findings to the MAST failure-mode taxonomy (Cemri et al., NeurIPS 2025). Candidates and evidence only — HOTL, never a verdict. → Evaluator_Examples/ch28_rca_diagnosis.py
• Statistically valid A/B testing — agent-eval abtest auto-selects Welch's t-test (2 files), mSPRT always-valid inference (--sequential, safe under repeated peeking), or N-way + FDR correction (3+ files). → Evaluator_Examples/ch29_sequential_ab_test.py
• Machine-readable insight layer + closed improvement loop — every result JSON carries an extra_metrics.insights object (deployment-readiness verdict, Path to Green, failure clustering, paste-ready fix snippets, per-(gate, change) track record). agent-eval target pins your project SLOs, agent-eval benchmark an external reference distribution, and agent-eval experiment / agent-eval improve register a hypothesis → apply → re-verify loop. Schema-validated, never raises. → Docs/09_OUTPUTS.md
• Real-time guardrail — two reference stacks — the same LiveGuardrail engine blocks a single tool call before it runs (Gate B/E), wired into either AOO (Agent-Evaluator + Ollama + OpenCode — fully local, no cloud model) via agent-eval opencode install, or AC (Agent-Evaluator + Claude Code — native CLI hooks) via agent-eval claude install. Identical verdict logic; the difference is the process model (a resident subprocess vs. per-call replay). A blocked call keeps a redacted excerpt of its command, so agent-eval {claude,opencode} violations / blocked-detail <task_id> (and the show_violation MCP tool) surface what was blocked in a later session. → Docs/AOO_STACK.md · Docs/CLAUDE_CODE_HOOKS.md · Docs/OPENCODE_VS_CLAUDE_CODE.md
• Dashboard — agent-eval dashboard (FastAPI): Harness Gate breakdown, File Compare with pairwise LLM Judge, anomaly/cost tracking, and a 🔧 Improve tab surfacing the RCA engine.

Installation

Extras are organized into 5 categories by intent — pick the one(s) that match what you're trying to do. Every category is additive and independent; combine as needed.

Single-feature extras that don't fit the 5 categories above: [export] (dashboard Parquet/Excel), [wandb], [mlflow]. Full package-by-package breakdown: pyproject.toml.

CLI Commands

Examples

32 standalone, book-chapter-based files in Evaluator_Examples/ (ch01–ch32), covering everything from a first evaluation to the full RCA/A/B-testing improvement loop:

pip install "agent-evaluator[examples]"
cd Evaluator_Examples && python ch01_first_eval.py   # ... through ch32_ollama_realtime.py

Project Structure

agent_evaluator/
├── decorators.py     # agent_eval · batch_eval · conversation_eval · QuickEval facade (quick_eval.py)
├── gates/            # Gate A–G scoring (gate_a_goal/ … gate_g_observability/) + LiveGuardrail
├── core/trackers/    # 25 Native Trackers (Layer 1 foundation · Layer 2 agentic/security) + monitor.py
├── rca/              # diagnose() — Gate-regression root-cause diagnosis + improvement/experiment logs
├── ontology/         # GATE_GUIDANCE · NATIVE_METRIC_RULES · MAST + single-agent failure taxonomies
├── reporting/        # insights.py (build_insights) + comprehensive_report.py (self-contained HTML)
├── integrations/     # LLMJudge · DeepEval/Ragas adapters · MCP servers · live-guardrail bridges
├── serve/            # FastAPI dashboard ([sdk] extra)
└── cli/              # agent-eval CLI (init, check, gate, diagnose, abtest, trend, dataset,
                      #   experiment, target, benchmark, improve, claims, monitor, opencode, claude)

Evaluator_Examples/   # 32 example files (ch01–ch32)
tests/                # 4,850+ test functions

Changelog

• v1.0.4 (2026-09-08) — Blocked-attempt command detail: a fully-blocked tool call keeps a redacted command excerpt, so search_violations "rm -rf" now matches the command itself; new show_violation MCP tool + agent-eval {claude,opencode} violations / blocked-detail <task_id> CLI; doctor live-checks it. Symmetric across Claude Code and OpenCode. No API/Config/schema changes.
• v1.0.3 (2026-09-08) — agent-eval claude install --with-violation-search now points the search_violations MCP server at the Claude Code batch-report DB (it opened the OpenCode default before); search_violations degrades to a readable sentence when the DB is missing. OpenCode unaffected.
• v1.0.2 (2026-09-04) — Phoenix / OTEL: Python-version-scoped arize-phoenix pin; span + trace + session annotation tiers; ASCII-only agent-eval monitor console; new Docs/10_OTEL_DATA_REFERENCE.md + local-Ollama example.
• v1.0.1 (2026-09-03) — Report-generation hardening (malformed / partial result JSON no longer crashes the report, gate, or the dashboard) + dashboard ↔ static-report value parity + English-only runtime output.
• v1.0.0 (2026-08-31) — General Availability: completes the machine-readable insight layer (extra_metrics.insights, ~62 schema-validated keys) + the target / benchmark / experiment / improve CLI loop.

Full history (incl. the 1.0.0-rc.1–rc4 series): CHANGELOG.md.

Documentation

Also available in-app once the dashboard is running: agent-eval dashboard → SDK Reference (/sdk-docs) and REST API (/api/docs).

Development

git clone https://github.com/bullpeng72/Agent-Evaluator.git
cd Agent-Evaluator
pip install -e ".[dev]"

pytest                          # run tests
ruff check agent_evaluator/    # lint
mypy agent_evaluator/          # type check

License

MIT — see LICENSE.