Agent Evaluator

Harness Engineering evaluation SDK that judges AI agent deployment readiness through 7 Gates.

It asks not just "does the agent work well?" but "is the agent ready for production?" One decorator line auto-recognizes 24 frameworks (LangChain, CrewAI, AutoGen, …) and measures 58 metrics (25 Native Trackers + 33 Harness Config) without touching your agent code — then aggregates them into 7 Gate pass/warn/fail judgments, a root-cause diagnosis engine for regressions, and statistically valid A/B testing.

pip install agent-evaluator

from agent_evaluator import QuickEval

eval = QuickEval("results/")

@eval.qa
def my_agent(question: str, ground_truth: str = "") -> str:
    return llm.invoke(question)          # your agent code — unchanged

my_agent("What is the capital of South Korea?", ground_truth="Seoul")

eval.save()                                        # results/quickeval.json + .html
eval.gate(tcr=85, accuracy=70, hallucination=5)    # CI/CD gate — sys.exit(1) if unmet

The 7 Harness Gates

Pass any of the 33 Configs above as @agent_eval/@batch_eval/@conversation_eval parameters and PerformanceMonitor auto-aggregates each Gate's pass/warn/fail from the underlying trackers — no separate scoring pass needed.

@agent_eval(monitor, task_type="qa",
    instructions=InstructionConfig(required_keywords=["Seoul"], fail_on_violation=True),   # Gate A
    loop_detection=LoopDetectionConfig(consecutive_repeat_threshold=6),                    # Gate B
    sla=SLAConfig(p95_ms=3000),                                                            # Gate D
)
def my_agent(question: str, ground_truth: str = "") -> str: ...

Full Gate reference: Docs/05_QUALITY_GATE.md · Runnable walkthrough: Evaluator_Examples/ch03_harness_basics.py

What's Inside

• 3 decorator types — @agent_eval (1 call → 1 result), @batch_eval (1 call → N results), @conversation_eval (N calls → 1 multi-turn result). All non-invasive: your function's signature, return value, and exceptions are untouched. → Docs/01_GETTING_STARTED.md
• 24 framework adapters — framework="langchain"/"crewai"/"anthropic"/"openai"/… auto-extracts tool_calls/chain_steps/tokens_used from the framework's native response object (duck typing — works without agent-evaluator importing the framework itself). → Docs/03_INTEGRATION_GUIDE.md
• 58 metrics — 25 Native Trackers (accuracy, hallucination, latency, tool efficiency, 5 security trackers, …) + the 33 Harness Configs above. → Docs/02_METRICS_GUIDE.md or the in-app SDK Reference (agent-eval dashboard → /sdk-docs)
• Self-contained HTML report — save_to_file() (and agent-eval gate --html-out) write one .html next to the JSON — no server, no external JS. It is laid out in 3 tiers: judgment (a one-line deployment-readiness verdict + a HIGH/MEDIUM/LOW confidence badge + Next actions 1·2·3 + a Path to Green — quantified gap to each failing gate, impact-ordered), iteration (a lifecycle-phase strip — analysis → operations — and a proof panel), and evidence (collapsible groups: per-gate Score Breakdown, worst failure cases each with a tool-call trajectory waterfall, the eval-set design contract, stats, versioning, governance). Pass a baseline and it adds the regressed/new/fixed failure-set diff plus prompt/config change attribution. agent-eval gate --html-summary prints a short Markdown block (verdict + path + the one next command) for a PR body. → Docs/13_OUTPUTS.md
• CI/CD quality gating — agent-eval gate result.json --tcr 85 --accuracy 70, plus baseline regression detection, per-version baselines, golden-set regression gating, --require-spec-coverage (exit 4 if a declared requirement has no golden case), and --hold-on-undecided (exit 75 — "hold for human" when the verdict is statistically borderline; never overrides a real fail). → Docs/05_QUALITY_GATE.md
• Root-cause diagnosis (RCA) — agent-eval diagnose / agent_evaluator.rca.diagnose() automates detect → attribute → cross-reference for a Gate regression, and links Gate F findings to the MAST failure-mode taxonomy (Cemri et al., 2025). Candidates and evidence only — HOTL, never a verdict. → Evaluator_Examples/ch28_rca_diagnosis.py
• Statistically valid A/B testing — agent-eval abtest auto-selects Welch's t-test (2 files), mSPRT always-valid inference (--sequential, safe under repeated peeking), or N-way + FDR correction (3+ files). → Evaluator_Examples/ch29_sequential_ab_test.py
• Machine-readable insight layer + closed improvement loop — every result JSON carries an extra_metrics.insights object (~65 keys: deployment-readiness verdict + decision_ready, Path to Green, failure clustering, paste-ready fix snippets, per-(gate, change) track record, lifecycle_phase, nondeterminism_repeat, eval_set_delta, spec_coverage, deploy_decision). agent-eval target pins your project SLOs, agent-eval benchmark an external reference distribution, and agent-eval experiment / agent-eval improve register a hypothesis → apply → re-verify loop (improve apply-verify runs the apply in a throw-away git worktree and never merges). Schema-validated, never raises. → Docs/13_OUTPUTS.md
• Development-support layer (all opt-in, no default behavior change) — create_taskresult(covers=[req_id]) ties golden cases to requirements; a deploy-decision ledger (gate --decision-log + agent-eval decisions record) records who accepted / held / overrode a gate run and why; FaultInjectionConfig injects seeded tool failures / latency into Gate C/D scoring without ever touching the blocking path; run_repeated() folds a K-run verdict-stability summary into insights.nondeterminism_repeat; a StreamingEvaluator(golden_candidate_sink=) queue routes production errors / low-confidence answers to agent-eval dataset review-candidates for human promotion. → Docs/05_QUALITY_GATE.md
• Real-time guardrail — two reference stacks, plus a host-less mode — the same LiveGuardrail engine blocks a single tool call before it runs (Gate B/E), wired into either AOO (Agent-Evaluator + Ollama + OpenCode — fully local, no cloud model) via agent-eval opencode install, or AC (Agent-Evaluator + Claude Code — native CLI hooks) via agent-eval claude install; identical verdict logic, the difference is the process model (a resident subprocess vs. per-call replay). No host at all? tool_guard() + live_guardrail_session() wrap any Python agent loop directly, and (1.1.0) audit_blocked=True is now the default, audit_log_path= flushes a durable record on exit even if the caller's own error handling is silent, and on_block=webhook_on_block(url) fires an out-of-band alert the instant a call is blocked. A blocked call keeps a redacted excerpt of its command, so agent-eval {claude,opencode} violations (with no query — 1.1.0 — browses recent history; a keyword narrows it) / blocked-detail <task_id> (and the list_violations / show_violation MCP tools) surface what was blocked; doctor now also reports the audit DB's row count proactively, and the HTML report shows blocked attempts directly (insights.blocked_attempts_audit) even though they never move Gate B/E scores. → Docs/06_LIVEGUARDRAIL.md · Docs/07_CLAUDE_CODE_HOOKS.md · Docs/08_AOO_STACK.md · Docs/09_OPENCODE_VS_CLAUDE_CODE.md
• Dashboard — agent-eval dashboard (FastAPI): Harness Gate breakdown, File Compare with pairwise LLM Judge, anomaly/cost tracking, and a 🔧 Improve tab surfacing the RCA engine.

Positioning & Direction

Where it fits

Agent-Evaluator is a deployment-readiness judge and iteration harness — not a tracing platform, and not a metric library:

Its center of gravity is where the ship / no-ship decision is made: the CI/CD gate and the analysis → design → development → verification loop before it. Streaming + OTEL cover production, but that is not the focus.

What makes it different

• Seven non-redundant Gates, not one score. Goal, behavior, reliability, performance, security, multi-agent coordination, observability — each graded pass / warn / fail. "83% accuracy" cannot hide a security regression, because they are scored separately.
• A verdict and a path, not just numbers. Every report and insights object leads with ship / hold / not-ready + a HIGH/MEDIUM/LOW confidence badge, then an impact-ordered Path to Green with paste-ready @agent_eval snippets — the smallest next step, not a dashboard to interpret.
• Two closed loops, deliberately separated. A real-time guardrail blocks a dangerous tool call before it runs (Gate B/E); a batch pass grades after. Different code, different timing — the guardrail never scores and the scorer never blocks (FaultInjectionConfig, --hold-on-undecided, everything respects that line).
• Statistical honesty over a confident number. Wilson CI on the pass-rate; decision_ready → exit 75 ("hold for human") when the verdict is borderline; always-valid A/B inference (safe under repeated peeking); K-run verdict-stability; Benjamini–Hochberg correction across slices. It declines to over-claim on n = 12.
• Human-on-the-loop by construction. RCA emits candidates + evidence, never a verdict. improve apply-verify runs the apply in a throw-away git worktree and never merges or commits. A deploy-decision ledger records who accepted / held / overrode each gate run, and why.
• Non-invasive and framework-agnostic. One decorator; your function's signature, return value, and exceptions are untouched. 24 framework adapters by pure duck typing — agent-evaluator never imports the framework itself.
• A local-first option. The AOO stack (Agent-Evaluator + Ollama + OpenCode) runs the entire loop with no cloud model and no token bill; AC (Agent-Evaluator + Claude Code) targets cloud models. Same verdict engine, different process model.
• Backed by a written methodology. Harness Methodology (a companion book) states the discipline — decision before code · prove, don't pass · block ≠ score · don't rebuild what already exists · size the model to the task · the human stays accountable — and this SDK is one reference implementation of it.

Direction

The trajectory from 1.0.0 has moved from "score the agent" toward "drive the whole build-and-improve loop, with the human's judgment on record" — and, most recently, toward making the real-time guardrail trustworthy even with no host process watching it:

• 1.0.0 — the machine-readable insights layer + the target / benchmark / experiment / improve CLI loop.
• 1.0.5 — Harness Methodology alignment (--hold-on-undecided, human_only_patterns, verdict-stability), a development-support framework (requirement → golden-case coverage, a deploy-decision ledger, thin fault injection, a production → golden candidate queue, improve apply-verify), and the HTML report re-cast as a methodology instrument — three tiers (judgment / iteration / evidence), a lifecycle-phase read, and --html-summary for a PR body.
• 1.1.0 — LiveGuardrail discovery/durability hardening: a blocked call is durable and discoverable by default now, even for a host-less custom agent loop with no Claude Code/OpenCode bridge and no visible message (tool_guard(audit_blocked=True) default, audit_log_path= crash-safe flush, on_block=webhook_on_block(...) out-of-band alert) — plus keyword-free violations/ list_violations browsing and an HTML report section for everyone, since a blocked attempt never moves Gate B/E scores and a clean scorecard alone would hide it.

Explicit non-goals — the boundary is a deliberate design choice, not a missing feature: it does not author or parse specs (EARS), does not provide a sandbox (E2B / Firecracker — that is the team's infrastructure), does not enforce which model tier a task runs on (that is the runtime's), and never auto-merges or auto-deploys.

Installation

Extras are organized into 5 categories by intent — pick the one(s) that match what you're trying to do. Every category is additive and independent; combine as needed.

Single-feature extras that don't fit the 5 categories above: [export] (dashboard Parquet/Excel), [wandb], [mlflow]. Full package-by-package breakdown: pyproject.toml.

CLI Commands

Examples

32 standalone, book-chapter-based files in Evaluator_Examples/ (ch01–ch32), covering everything from a first evaluation to the full RCA/A/B-testing improvement loop:

pip install "agent-evaluator[examples]"
cd Evaluator_Examples && python ch01_first_eval.py   # ... through ch32_ollama_realtime.py

Project Structure

agent_evaluator/
├── decorators.py     # agent_eval · batch_eval · conversation_eval · QuickEval facade (quick_eval.py)
├── gates/            # Gate A–G scoring (gate_a_goal/ … gate_g_observability/) + LiveGuardrail
├── core/trackers/    # 25 Native Trackers (Layer 1 foundation · Layer 2 agentic/security) + monitor.py
├── rca/              # diagnose() — Gate-regression root-cause diagnosis + improvement/experiment logs
├── ontology/         # GATE_GUIDANCE · NATIVE_METRIC_RULES · MAST + single-agent failure taxonomies
├── reporting/        # insights.py (build_insights) + comprehensive_report.py (self-contained HTML)
├── integrations/     # LLMJudge · DeepEval/Ragas adapters · MCP servers · live-guardrail bridges
├── serve/            # FastAPI dashboard ([sdk] extra)
└── cli/              # agent-eval CLI (init, check, gate, decisions, diagnose, abtest, trend, dataset,
                      #   feedback, experiment, target, benchmark, improve, claims, monitor, opencode, claude)

Evaluator_Examples/   # 32 example files (ch01–ch32)
tests/                # 5,150+ test functions

Changelog

• v1.1.0 (2026-09-11) — LiveGuardrail discovery/durability hardening. tool_guard(audit_blocked=True) now default; on_block=/webhook_on_block() out-of-band block alerts; keyword-free list_violations() / violations browsing; blocked_attempt_capture.max_chars 240→500; insights.blocked_attempts_audit HTML section. Additive/opt-in beyond the default change.
• v1.0.6 (2026-09-10) — Maintenance: decorators.py split into framework_adapters.py + _eval_shared.py (re-exported, no API change); --help now lists all 18 subcommands.
• v1.0.5 (2026-09-09) — Harness Methodology alignment + dev-support framework + 3-tier HTML report. --hold-on-undecided, --require-spec-coverage, deploy-decision ledger, run_repeated(), FaultInjectionConfig, dataset review-candidates, improve apply-verify. All opt-in.
• v1.0.4 (2026-09-08) — Blocked calls keep a redacted command excerpt; new show_violation MCP tool + {claude,opencode} violations/blocked-detail CLI.
• v1.0.3 (2026-09-08) — Fixed search_violations MCP to point at the Claude Code DB (was defaulting to OpenCode's).
• v1.0.2 (2026-09-04) — Phoenix/OTEL: version-scoped pin, span/trace/session annotations, ASCII console output.
• v1.0.1 (2026-09-03) — Report-generation hardening (malformed JSON no longer crashes), dashboard/report value parity, English-only runtime output.
• v1.0.0 (2026-08-31) — General Availability: machine-readable insights layer (~62 keys) + target/benchmark/experiment/improve CLI loop.

Full history (incl. the 1.0.0-rc.1–rc4 series): CHANGELOG.md.

Documentation

Also available in-app once the dashboard is running: agent-eval dashboard → SDK Reference (/sdk-docs) and REST API (/api/docs).

Development

git clone https://github.com/bullpeng72/Agent-Evaluator.git
cd Agent-Evaluator
pip install -e ".[dev]"

pytest                          # run tests
ruff check agent_evaluator/    # lint
mypy agent_evaluator/          # type check

License

MIT — see LICENSE.