AgentEval 🧪
Testing and evaluation framework for AI agents. Define test suites in YAML, grade agent outputs with 10 pluggable graders, track results over time, and detect regressions with statistical comparison.
Why AgentEval?
AI agents are hard to test. They're non-deterministic, they call tools, and their outputs vary between runs. Traditional unit tests don't cut it.
- 🎯 YAML-based test suites — Define inputs, expected outputs, and grading criteria declaratively
- 📊 Statistical regression detection — Welch's t-test across multiple runs, not just pass/fail
- 🔌 10 built-in graders — Exact match, contains, regex, tool-check, LLM-judge, custom, JSON-schema, semantic, latency, and cost
- 🔗 AgentLens integration — Import production sessions as test cases, and emit eval runs back as tamper-evident audit evidence
- 💰 Cost & latency tracking — Know what each eval costs in tokens and dollars
- 🗄️ SQLite result storage — Every run is persisted for historical comparison
Quick Start
pip install agentevalkit
1. Define a test suite
# suite.yaml
name: my-agent-tests
agent: my_agent:run
cases:
- name: basic-math
input: "What is 2 + 2?"
expected:
output_contains: ["4"]
grader: contains
- name: tool-usage
input: "Search for the weather in NYC"
expected:
tools_called: ["web_search"]
grader: tool-check
- name: format-check
input: "List 3 colors"
expected:
pattern: "\d\.\s+\w+"
grader: regex
2. Create your agent callable
# my_agent.py
from agenteval.models import AgentResult
def run(input_text: str) -> AgentResult:
# Your agent logic here
return AgentResult(
output="The answer is 4.",
tools_called=[{"name": "web_search", "args": {"query": "weather NYC"}}],
tokens_in=12,
tokens_out=8,
cost_usd=0.0003,
)
3. Run the eval
$ agenteval run --suite suite.yaml --verbose
============================================================
Suite: my-agent-tests | Run: c1c6493118d5
============================================================
PASS basic-addition (score=1.00, 150ms)
PASS capital-city (score=1.00, 200ms)
PASS quantum-summary (score=1.00, 350ms)
PASS tool-usage (score=1.00, 280ms)
PASS list-format (score=1.00, 120ms)
Total: 5 Passed: 5 Failed: 0 Pass rate: 100%
Cost: $0.0023 Avg latency: 220ms
Features
🎯 10 Built-in Graders
| Grader | What it checks | Expected / config fields |
|---|---|---|
exact |
Exact string match | output |
contains |
Substring presence | output_contains: [list] |
regex |
Pattern matching | pattern |
tool-check |
Tools were called | tools_called: [list] |
llm-judge |
LLM evaluates quality | criteria (free-form) |
custom |
Your own function | grader_config: {function: "mod:fn"} |
json_schema |
Output validates against a JSON Schema | grader_config: {schema: {...}} or {schema_file: path} |
semantic |
Cosine similarity to expected text | grader_config: {expected: str, threshold: 0.8} |
latency |
Response time within budget | grader_config: {max_ms: N} |
cost |
Cost within budget | grader_config: {max_usd: N} |
📊 Statistical Comparison
Compare runs with Welch's t-test to detect statistically significant regressions:
$ agenteval compare c1c6493118d5,d17a2dce0222 4ee7e40601e3,ba5b0dde212b
============================================================================
Comparing: c1c6493118d5,d17a2dce0222 vs 4ee7e40601e3,ba5b0dde212b
Alpha: 0.05 Regression threshold: 0.0
============================================================================
Case Base Target Diff p-value Sig Status
----------------------------------------------------------------------------
basic-addition 1.000 1.000 +0.000 —
capital-city 1.000 0.500 -0.500 0.4533
quantum-summary 1.000 0.500 -0.500 0.4533
tool-usage 1.000 0.000 -1.000 0.0000 * ▼ regressed
list-format 1.000 0.500 -0.500 0.4533
Summary: 0 improved, 1 regressed, 4 unchanged
⚠ 1 regression(s) detected!
Run the same suite multiple times and compare groups: agenteval compare RUN_A1,RUN_A2 vs RUN_B1,RUN_B2. Uses scipy when available, falls back to pure Python.
🔗 AgentLens Integration
Import real agent sessions from AgentLens as test suites:
# From AgentLens SQLite database
agenteval import --from agentlens --db sessions.db --output suite.yaml --grader contains
# From AgentLens server API (single session or --batch)
agenteval import-agentlens --server http://localhost:3000 --session SESSION_ID --output suite.yaml
agenteval import-agentlens --server http://localhost:3000 --batch --limit 100 --output suite.yaml
# With filtering and interactive review (server mode)
agenteval import-agentlens --server http://localhost:3000 --batch --filter-tag production --auto-assertions --interactive --output suite.yaml
Import modes:
- SQLite mode (
import --from agentlens --db path) — reads directly from an AgentLens database file - Server mode (
import-agentlens --server URL) — fetches sessions via the AgentLens HTTP API (use--session IDfor one session or--batchfor many)
Sessions are converted to eval cases with input/output mapping and optional tool-call assertions. Use --auto-assertions to automatically generate expected fields from session data, and --interactive to review each case before saving.
Turn production traffic into regression tests — no manual test writing needed.
Emit eval runs back to AgentLens (the reverse direction): add --agentlens-server to agenteval run and the completed run is recorded in AgentLens as a server-authoritative, hash-chained eval_result — tamper-evident eval evidence on the audit trail.
# Record this run in AgentLens (best-effort: a federation hiccup never fails the run)
agenteval run --suite suite.yaml \
--agentlens-server https://agentlens.example \
--agentlens-token "$AGENTLENS_SERVICE_TOKEN"
- Authenticates with AgentLens's service token (the federation endpoint is service-token gated, not API-key gated) —
--agentlens-tokenorAGENTLENS_SERVICE_TOKEN. All flags also read env vars:AGENTLENS_SERVER,AGENTLENS_TENANT_ID. - By default the result chains into a synthetic per-run session (
eval-<run id>); pass--agentlens-session-idto chain it into an existing instrumented-agent session instead. - The run's pass-rate becomes the score, failed cases become violations, and
eval_resultstays server-authoritative — agenteval can't forge evidence, only the server emits it.
💰 Cost & Latency Tracking
Every eval tracks tokens and cost. Your agent callable returns AgentResult with tokens_in, tokens_out, and cost_usd, and AgentEval aggregates them per run.
YAML Suite Format
Full annotated example:
name: my-agent-tests # Suite name (shown in reports)
agent: my_module:my_agent # Default agent callable (module:function)
defaults: # Defaults applied to all cases
grader: contains
grader_config:
ignore_case: true
cases:
- name: basic-math # Unique case name
input: "What is 2 + 2?" # Input passed to agent
expected: # Grader-specific expected values
output_contains: ["4"]
grader: contains # Override default grader
tags: [math, basic] # Tags for filtering (--tag math)
- name: tool-usage
input: "Search for weather"
expected:
tools_called: ["web_search"]
grader: tool-check
- name: quality-check
input: "Explain gravity"
expected:
criteria: "Should mention Newton or Einstein, be scientifically accurate"
grader: llm-judge
grader_config:
model: gpt-4o-mini # LLM judge model
api_base: https://api.openai.com/v1
- name: custom-validation
input: "Generate a JSON object"
expected: {}
grader: custom
grader_config:
function: my_graders:validate_json # Your grader function
CLI Reference
agenteval run
agenteval run --suite suite.yaml [--agent module:fn] [--verbose] [--tag math] [--timeout 30] [--db agenteval.db]
--suite— Path to YAML suite file (required)--agent— Override the agent callable from the suite--verbose/-v— Show per-case pass/fail details--tag— Filter cases by tag (repeatable)--timeout— Per-case timeout in seconds (default: 30)--db— SQLite database path (default:agenteval.db)
Exit code is 1 if any case fails.
agenteval list
agenteval list [--suite-filter name] [--limit 20] [--db agenteval.db]
$ agenteval list --limit 5
ID Suite Passed Failed Rate Created
--------------------------------------------------------------------------------
aeccd5e53f03 math-agent-demo 2 3 40% 2026-02-12T21:12:12
4f3e380f622c math-agent-demo 3 2 60% 2026-02-12T21:12:12
bd4ef3a0727b math-agent-demo 1 4 20% 2026-02-12T21:12:12
e2ca43e99852 math-agent-demo 3 2 60% 2026-02-12T21:12:11
32ed650cab6d math-agent-demo 2 3 40% 2026-02-12T21:12:11
agenteval compare
agenteval compare RUN_A RUN_B [--alpha 0.05] [--threshold 0.0] [--stats/--no-stats]
agenteval compare RUN_A1,RUN_A2 vs RUN_B1,RUN_B2 # Multi-run comparison
agenteval import
agenteval import --from agentlens --db sessions.db --output suite.yaml [--grader contains] [--limit 100]
Grader Reference
exact
Compares result.output exactly with expected.output. Config: ignore_case: bool.
expected:
output: "The answer is 42."
grader: exact
grader_config:
ignore_case: true
contains
Checks that all substrings in expected.output_contains appear in the output.
expected:
output_contains: ["Paris", "France"]
grader: contains
regex
Matches result.output against expected.pattern (Python regex). Config: flags: [IGNORECASE, DOTALL, MULTILINE].
expected:
pattern: "\d+\.\d+"
grader: regex
grader_config:
flags: [IGNORECASE]
tool-check
Verifies expected tools were called. Config: ordered: bool for sequence matching.
expected:
tools_called: ["web_search", "calculator"]
grader: tool-check
grader_config:
ordered: true
llm-judge
Sends the input, output, and criteria to an LLM for evaluation. Requires OPENAI_API_KEY or compatible API.
expected:
criteria: "Response should be helpful, accurate, and concise"
grader: llm-judge
grader_config:
model: gpt-4o-mini
custom
Imports and calls your own grader function. Must accept (case: EvalCase, result: AgentResult) -> GradeResult.
grader: custom
grader_config:
function: my_module:my_grader
Adapters
Adapters let you test agents built with popular frameworks without writing a custom callable.
pip install agentevalkit[langchain] # LangChain
pip install agentevalkit[crewai] # CrewAI
pip install agentevalkit[autogen] # AutoGen
| Adapter | Framework Method | Install Extra |
|---|---|---|
langchain |
agent.invoke(input) |
[langchain] |
crewai |
crew.kickoff(inputs={"input": ...}) |
[crewai] |
autogen |
agent.run(input) or agent.initiate_chat(message=...) |
[autogen] |
Usage with YAML suite defaults:
# suite.yaml
name: my-tests
agent: my_module:my_chain
defaults:
adapter: langchain
Or via CLI:
agenteval run --suite suite.yaml --adapter langchain
Each adapter extracts output, tool calls, and token usage from the framework's response format into a standard AgentResult.
Distributed Execution
Scale eval suites across multiple workers using Redis as a broker.
Setup
pip install agentevalkit[distributed]
Start Workers
# Terminal 1: Start a worker
agenteval worker --broker redis://localhost:6379 --agent my_module:my_agent
# Terminal 2: Start another worker
agenteval worker --broker redis://localhost:6379 --agent my_module:my_agent
Run with Workers
agenteval run --suite suite.yaml --workers redis://localhost:6379 --worker-timeout 60
How It Works
- The coordinator pushes eval cases to a Redis queue
- Workers pop cases, execute the agent, and push results back
- The coordinator collects results and builds the final
EvalRun - If no workers are detected, execution falls back to local mode automatically
Configuration
--workers URL— Redis broker URL (supportsredis://andrediss://for TLS)--worker-timeout N— Seconds to wait for worker results (default: 30)- Workers register heartbeats and are automatically detected by the coordinator
Security: Use
rediss://URLs with authentication for production deployments. See docs/troubleshooting.md for Redis security guidance.
Troubleshooting
See docs/troubleshooting.md for solutions to common issues including:
- Agent callable import errors (
module:functionformat) - Missing dependency extras (
[distributed],[langchain], etc.) - OpenAI API key setup for
llm-judgegrader - Compare command syntax
- Redis connection issues for distributed execution
Contributing
Contributions welcome! This project uses:
- pytest for testing
- ruff for linting
- src layout (
src/agenteval/)
git clone https://github.com/agentkitai/agenteval.git
cd agenteval
pip install -e ".[dev]"
pytest
🧰 AgentKit Ecosystem
| Project | Description | |
|---|---|---|
| AgentLens | Observability & audit trail for AI agents | |
| Lore | Cross-agent memory and lesson sharing | |
| AgentGate | Human-in-the-loop approval gateway | |
| FormBridge | Agent-human mixed-mode forms | |
| AgentEval | Testing & evaluation framework | ⬅️ you are here |
| agentkit-cli | Unified CLI orchestrator |
License
MIT — see LICENSE.
Metadata
Release files for agentevalkit 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentevalkit-0.7.0.tar.gz | 120.0 kB | With provenance | Details for agentevalkit-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentevalkit-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for agentevalkit-0.7.0-py3-none-any.whl |
Total release size: 228.1 kB
Release files / agentevalkit-0.7.0.tar.gz
| Download URL | agentevalkit-0.7.0.tar.gz |
|---|---|
| Size | 120.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a184c6224758efd8186b5804c780cadb4948c189045387d08e03a21655cd6d3
|
|
BLAKE2b-256 checksum How to use checksums |
89651b5325a3debf5b80819c3e97cc1c6eb562fdc6a90f2f341c6ae83c0ec10f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 25, 2026.
Transparency logRelease files / agentevalkit-0.7.0-py3-none-any.whl
| Download URL | agentevalkit-0.7.0-py3-none-any.whl |
|---|---|
| Size | 108.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
eed8c93affc5a16200bd43662d6fd3b0963b00dd25dbc659de62f5d2516d3288
|
|
BLAKE2b-256 checksum How to use checksums |
86c1a09c523603ae01b8f6aee74bffc770a53b6317f28cdbaa50017091ac3f6e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 25, 2026.
Transparency logSecurity for agentevalkit 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for agentevalkit-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 25, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.