AgentEval

CI for AI agents — turn flaky agent behavior and production failures into tests that fail the PR.

Static demo · Streamlit dashboard · PyPI: nishanttyagi-agenteval · CLI: agenteval

The problem

You change a prompt, model, or tool. The agent still “answers.” Unit tests still pass. A week later someone finds a wrong refund, a hallucinated fact, or a tool the agent never should have called.

Normal tests prove the code ran. They do not prove the agent still behaves.

In simple terms

Imagine a support agent gets:

“Cancel order 4821 and refund the customer.”

A green unit test only means something came back. AgentEval checks the behavior:

If the agent says “refunded” without calling the refund tool, that can look fine to a human skim. AgentEval records it as a failure and can block the PR.

What I built

Four outcomes, not a platform pitch:

1. A regression gate for agents — YAML golden suites, scored runs, compare to a git-trackable baseline, fail CI when quality drops.
2. Evidence, not one score — correctness, hallucination, tool-call accuracy, latency/cost, trajectory, flakiness, plus optional RAG checks and a SQL safety scanner.
3. Failure Memory — capture → redact → cluster → replay → minimize → human approve → golden YAML → CI, so the same production failure is harder to ship twice.
4. One CLI across frameworks — adapters for CrewAI, AutoGen, OpenAI Agents SDK, LangGraph, and custom agents; composite GitHub Action; templates (including an Indic-language pack).

A few numbers

Evidence from this repo on main — no invented percentages.

Why this matters

If you ship agent changes, you want (1) a hard gate when known behavior regresses, and (2) a memory of real failures that does not reset every run. AgentEval is built for that workflow: golden YAML and baselines live in git; Failure Memory stays local-first with human approval before anything blocks CI; demos run without network or provider keys.

How it works

Simple flow

1. Write expected behavior as YAML golden cases (or scaffold with agenteval init).
2. Run the agent through an adapter: agenteval run.
3. Compare to a baseline: agenteval compare (CI fails on regression).
4. Optionally: ingest a production failure into Failure Memory → redact → replay → minimize → approve → export golden → run with --production-cases.
5. Inspect evidence: JSON/HTML report, Streamlit dashboard, or agenteval trace / diff.

Optional architecture (for engineers)

Agent adapter + golden YAML  →  runner / evaluators  →  JSON run + HTML report
                                                      →  baseline compare / CI gate

Production traces  →  redact  →  Failure Memory (SQLite)
                              →  cluster / replay / minimize
                              →  human approve  →  golden YAML  →  same CI gate

Implemented in-repo: adapters/, core/, evaluators/, failure_memory/, CLI entry agenteval → agenteval.cli:main, optional Streamlit dashboard, composite Action (action.yml).

Install

Python 3.10+.

From PyPI (latest published release is 0.3.0):

pip install nishanttyagi-agenteval==0.3.0
agenteval --version
agenteval --help

From this repo (current main is 0.5.0):

git clone https://github.com/nishanttyagi28/agenteval.git
cd agenteval
python -m pip install -e ".[dev]"
agenteval --version   # expect 0.5.0 on main
python -m pytest -q

Framework extras (optional): pip install "nishanttyagi-agenteval[crewai]" (same for autogen, openai-agents, karmasakshi).

Quick start

Offline mock agent (no provider)

agenteval run --agent mock_agent --registry examples/mock_agent/agents.yaml
agenteval compare --agent mock_agent --registry examples/mock_agent/agents.yaml

Failure Memory demo (zero network, no API key)

python examples/failure_memory_demo_v21/run_demo.py

Capture → redact → cluster → replay → minimize → export golden → fail broken agent / pass fixed agent → coverage check. See examples/failure_memory_demo_v21/ and docs/failure-memory.md.

KarmaSakshi bridge demo (offline seal → witness → score)

pip install -e ".[dev,karmasakshi]"
python examples/karmasakshi_bridge/run_demo.py

Approved ₹1500→Priya: correct attempt passes; ₹1501 or wrong payee is blocked by KarmaSakshi and recorded as an AgentEval failure. See examples/karmasakshi_bridge/ and docs/karmasakshi-bridge.md.

Scaffold a project

agenteval init

Release Desk (v0.5.0)

Turn a production miss into a CI test without leaving the browser. No API key.

agenteval gate --local

Open the URL it prints (default http://127.0.0.1:8741). Load the refund sample, or POST an incident:

curl -s -X POST http://127.0.0.1:8741/api/gate/ingest \
  -H 'Content-Type: application/json' \
  -d '{"incident":{"agent":"refund-desk","prompt":"Refund order 4821","output":"Cancelled. No refund.","tools_called":["cancel_order"],"expected_tools":["lookup_order","issue_refund"]}}'

Ship writes .agenteval/production-regressions.yaml. The next run that still does this fails:

agenteval run --agent my_agent --production-cases .agenteval/production-regressions.yaml

The desk is local-only (pass --local). It redacts secret-shaped strings before storage and will not bind a public address unless you pass --allow-remote. See docs/release-desk.md.

Open-ended llm_judge cases no longer require the data-analyst sibling repo. With no key, the deterministic offline judge scores them. Set AGENTEVAL_JUDGE_PROVIDER to openai, groq, anthropic, or legacy.

Indic-language pack (v0.4.0 on main)

pip install -e examples/plugins/agenteval-indic-evaluators
agenteval run --agent indic_mock_agent \
  --registry examples/indic_mock_agent/agents.yaml \
  --tag core --no-llm-judge

34 cases (28 deterministic offline; 6 refusal/safety need an LLM judge). The mock demo expects deliberate failures so each checker is shown catching something — see examples/indic_mock_agent/README.md.

CLI

agenteval run | compare | report | generate | generate-adversarial | import | generate-cases
agenteval init | compare-models | trace | diff | calibrate | audit-log | serve
agenteval plugins | templates | sql | memory | gate

Failure Memory CLI

agenteval memory init
agenteval memory ingest traces.jsonl
agenteval memory cluster
agenteval memory list
agenteval memory review <candidate_id> approve --correctness-type contains --ground-truth "…"
agenteval memory replay <candidate_id>
agenteval memory minimize <candidate_id>
agenteval memory approve-minimization <minimization_id>
agenteval memory export-minimized <minimization_id>
agenteval memory coverage
agenteval run --agent my_agent --production-cases .agenteval/production-regressions.yaml

Default DB: .agenteval/failure-memory.db (--db or AGENTEVAL_FAILURE_MEMORY_DB).

Production failure → redact → ingest → cluster → replay → minimize
  → human approve → golden YAML → CI gate → recurrence / coverage

What AgentEval evaluates

Framework registry example

version: 1
agents:
  mock_agent:
    adapter: examples.mock_agent.adapter:MockAgentAdapter
    cases: examples/mock_agent/cases.yaml
    enabled: true

Composite Action for consumer repos:

- uses: nishanttyagi28/agenteval@v0.3.0
  with:
    agent: my_agent
    config-file: agents.yaml
    cases-file: tests/golden/cases.yaml
    baseline-file: baselines/my_agent.json

Pin a release tag you trust. main moves; PyPI 0.5.0 is not published yet. The latest published release remains 0.3.0.

Security and privacy defaults

Limitations and non-goals

• Local-first / single-user Failure Memory — not a hosted multi-tenant product
• Not an OpenTelemetry collector (optional OTel-shaped JSON helpers only)
• No mandatory vector DB or embeddings for clustering
• No mandatory LLM judge for classification / clustering
• Redaction will miss arbitrary unlabeled secrets
• Live agent eval still needs your runtime and any provider keys you choose
• Cost may fall back to estimates when providers omit usage
• Adversarial / generated cases stay candidates until a human promotes them
• Not claiming v1 stability — see docs/v1-readiness.md

Documentation

Status

WIP · Alpha · main at 0.5.0 · PyPI latest published 0.3.0. Useful today for local eval loops, golden suites, the Release Desk, and CI experiments. APIs and schemas can still move — pin a commit or release tag if you depend on behavior. Not a hosted observability replacement.

Why I built this

I kept hitting the same gap: agent quality lived in manual spot-checks and chat paste, while the rest of the stack had real CI. Plausible answers hid wrong tools, flaky trajectories, and regressions that only showed up after ship. Then the same production failure would return because nothing turned it into a permanent test.

I built AgentEval to make agent regression gates and failure memory as boring as unit tests — CLI-first, local-first, human approval before anything blocks merge — starting from a builder’s harness rather than a product pitch.

License

MIT. See LICENSE.