Agent Evaluator

Harness Engineering evaluation SDK that judges AI agent deployment readiness through 7 Gates.

It asks not just "does the agent work well?" but "is the agent ready for production?" One decorator line auto-recognizes 24 frameworks (LangChain, CrewAI, AutoGen, …) and measures 58 metrics (25 Native Trackers + 33 Harness Config) without touching your agent code — then aggregates them into 7 Gate pass/warn/fail judgments, a root-cause diagnosis engine for regressions, and statistically valid A/B testing.

pip install agent-evaluator

from agent_evaluator import QuickEval

eval = QuickEval("results/")

@eval.qa
def my_agent(question: str, ground_truth: str = "") -> str:
    return llm.invoke(question)          # your agent code — unchanged

my_agent("What is the capital of South Korea?", ground_truth="Seoul")

eval.save()                                        # results/quickeval.json + .html
eval.gate(tcr=85, accuracy=70, hallucination=5)    # CI/CD gate — sys.exit(1) if unmet

The 7 Harness Gates

Pass any of the 33 Configs above as @agent_eval/@batch_eval/@conversation_eval parameters and PerformanceMonitor auto-aggregates each Gate's pass/warn/fail from the underlying trackers — no separate scoring pass needed.

@agent_eval(monitor, task_type="qa",
    instructions=InstructionConfig(required_keywords=["Seoul"], fail_on_violation=True),   # Gate A
    loop_detection=LoopDetectionConfig(consecutive_repeat_threshold=6),                    # Gate B
    sla=SLAConfig(p95_ms=3000),                                                            # Gate D
)
def my_agent(question: str, ground_truth: str = "") -> str: ...

Full Gate reference: Docs/05_QUALITY_GATE.md · Runnable walkthrough: Evaluator_Examples/ch03_harness_basics.py

What's Inside

• 3 decorator types — @agent_eval (1 call → 1 result), @batch_eval (1 call → N results), @conversation_eval (N calls → 1 multi-turn result). All non-invasive: your function's signature, return value, and exceptions are untouched. → Docs/01_GETTING_STARTED.md
• 24 framework adapters — framework="langchain"/"crewai"/"anthropic"/"openai"/… auto-extracts tool_calls/chain_steps/tokens_used from the framework's native response object (duck typing — works without agent-evaluator importing the framework itself). → Docs/03_INTEGRATION_GUIDE.md
• 58 metrics — 25 Native Trackers (accuracy, hallucination, latency, tool efficiency, 5 security trackers, …) + the 33 Harness Configs above. → Docs/02_METRICS_GUIDE.md or the in-app SDK Reference (agent-eval dashboard → /sdk-docs)
• CI/CD quality gating — agent-eval gate result.json --tcr 85 --accuracy 70, plus baseline regression detection, per-version baselines, and golden-set regression gating. → Docs/05_QUALITY_GATE.md
• Root-cause diagnosis (RCA) — agent-eval diagnose / agent_evaluator.rca.diagnose() automates detect → attribute → cross-reference for a Gate regression, and links Gate F findings to the MAST failure-mode taxonomy (Cemri et al., NeurIPS 2025). Candidates and evidence only — HOTL, never a verdict. → Evaluator_Examples/ch28_rca_diagnosis.py
• Statistically valid A/B testing — agent-eval abtest auto-selects Welch's t-test (2 files), mSPRT always-valid inference (--sequential, safe under repeated peeking), or N-way + FDR correction (3+ files). → Evaluator_Examples/ch29_sequential_ab_test.py
• Real-time guardrail (AOO stack) — LiveGuardrail blocks a single tool call before it executes (Gate B/E), with a reference OpenCode plugin (agent-eval opencode install) and native Claude Code CLI hooks (agent-eval claude install). → Docs/AOO_STACK.md · Docs/CLAUDE_CODE_HOOKS.md
• Dashboard — agent-eval dashboard (FastAPI): Harness Gate breakdown, File Compare with pairwise LLM Judge, anomaly/cost tracking, and a 🔧 Improve tab surfacing the RCA engine.

Installation

Extras are organized into 5 categories by intent — pick the one(s) that match what you're trying to do. Every category is additive and independent; combine as needed.

Single-feature extras that don't fit the 5 categories above: [export] (dashboard Parquet/Excel), [wandb], [mlflow]. Full package-by-package breakdown: pyproject.toml.

CLI Commands

Examples

31 standalone, book-chapter-based files in Evaluator_Examples/ (ch01–ch31), covering everything from a first evaluation to the full RCA/A/B-testing improvement loop:

pip install "agent-evaluator[examples]"
cd Evaluator_Examples && python ch01_first_eval.py   # ... through ch31_recommendation_tracking.py

Project Structure

agent_evaluator/
├── decorators.py       # agent_eval · batch_eval · conversation_eval · QuickEval facade (quick_eval.py)
├── gates/               # Gate A–G scoring (gate_a_goal/ … gate_g_observability/) + LiveGuardrail
├── core/trackers/       # 25 Native Trackers (Layer 1 foundation · Layer 2 agentic/security) + monitor.py
├── rca/                  # diagnose() — Gate regression root-cause diagnosis + recommendation tracking
├── ontology/              # GATE_GUIDANCE / MAST failure-mode taxonomy (Gate F)
├── integrations/         # LLMJudge · DeepEval/Ragas adapters · MCP servers
├── serve/                # FastAPI dashboard ([serve] extra)
├── cli/                  # agent-eval CLI (gate, diagnose, abtest, trend, claims, opencode, claude, monitor, dataset)
└── reporting/             # comprehensive_report.py — self-contained HTML report generation

Evaluator_Examples/       # 31 example files (ch01–ch31)
tests/                    # 3,900+ test functions

Changelog

v1.0.0-rc.1 (2026-08-27) — Release candidate for v1.0.0.

Beta version history: CHANGELOG.md

Documentation

Also available in-app once the dashboard is running: agent-eval dashboard → SDK Reference (/sdk-docs) and REST API (/api/docs).

Development

git clone https://github.com/bullpeng72/Agent-Evaluator.git
cd Agent-Evaluator
pip install -e ".[dev]"

pytest                          # run tests
ruff check agent_evaluator/    # lint
mypy agent_evaluator/          # type check

License

MIT — see LICENSE.