AgentVerdict

Benchmark and compare AI agents with reproducible tests.

AgentVerdict is an open-source Python CLI for testing agent behavior across deterministic benchmarks, repeated runs, and side-by-side comparisons.

pip install agentverdict
agentverdict test my_agent.py

Quickstart

AgentVerdict currently loads agents through a small Python contract. Create my_agent.py:

def run_agent(task: str) -> str:
    return f"Received: {task}"

Run it against the bundled benchmark:

agentverdict test my_agent.py
agentverdict test my_agent.py --runs 5

For a representative passing run, use the included example:

agentverdict test examples/benchmark_agent.py --runs 5

AgentVerdict

Agent: benchmark_agent
Benchmark: General Agent Benchmark
Runs per task: 5

Tasks:                  10
Runs per task:           5
Total attempts:          50
Passed attempts:         50
Failed attempts:         0
Execution errors:        0
Overall reliability:     100.0%
Average consistency:     100.0%
Error rate:              0.0%
Average latency:          0.000s

The exact latency depends on your machine and the agent being tested.

Why AgentVerdict?

An AI agent can work in a demo and still behave differently across repeated runs. AgentVerdict measures the same agent against the same deterministic tasks so you can inspect:

• correctness
• reliability across attempts
• literal output consistency
• execution errors
• local execution latency

Compare agents

Run two or more agents against the same benchmark:

agentverdict compare \
    examples/benchmark_agent.py \
    examples/simple_agent.py \
    --runs 5

Each agent receives the same benchmark, task order, evaluator semantics, and runs per task. The comparison presents the individual metrics rather than inventing an arbitrary weighted overall score, so tradeoffs remain visible.

Benchmark format

Benchmarks are versioned YAML files. A minimal definition looks like this:

version: "1"
name: Example Benchmark

tasks:
  - id: capital-france
    name: Capital of France
    category: factual
    prompt: "Name the capital of France."
    evaluation:
      type: contains
      expected: "Paris"

Current deterministic evaluators are exact_match, contains, and regex. Use agentverdict benchmark-info to inspect a bundled benchmark or YAML file. The result schema documentation describes the structured output format.

Metrics

Reliability is passed attempts divided by total attempts.

Output consistency is, for each task, the frequency of its most common valid literal output, averaged across tasks. It is case-sensitive, whitespace-sensitive, deterministic, and not semantic similarity. Execution errors are excluded because they have no output.

Error rate is execution errors divided by total attempts.

Latency is measured locally for each agent execution. It reflects the machine, environment, and any services contacted by the agent.

PASS, FAIL, and ERROR

• PASS — the agent returned a valid output and satisfied the evaluator.
• FAIL — the agent executed successfully but did not satisfy the evaluator.
• ERROR — the agent failed to produce a valid output.

Reports

Generate reports locally while testing or comparing agents:

agentverdict test agent.py --runs 5 --report html
agentverdict compare agent_a.py agent_b.py --report all

--report accepts none, json, html, or all. HTML reports are standalone and require no server. JSON reports are useful for tooling and CI. Reports are generated locally; AgentVerdict does not upload them.

CLI reference

Command	Purpose
agentverdict run	Run a Python-file agent once.
agentverdict test	Test one agent against a benchmark.
agentverdict compare	Compare two or more agents on one benchmark.
agentverdict benchmarks	List bundled benchmarks.
agentverdict benchmark-info	Inspect a benchmark definition.
agentverdict --version	Show the installed version.

For detailed options, run agentverdict COMMAND --help.

CI usage

agentverdict test agent.py can run directly in CI.

• test: exit 0 when every attempt passes, 1 when the benchmark completes with failures or errors, and 2 for a configuration or application error.
• compare: exit 0 when comparison completes and 2 for a configuration or application error.

Installation and requirements

AgentVerdict requires Python 3.11 or newer.

For the public release:

pip install agentverdict

For local development:

python -m pip install -e ".[dev]"

Architecture

Agent
  ↓
AgentAdapter
  ↓
Benchmark Runner
  ↓
Evaluator
  ↓
Structured Results
  ├── Terminal
  ├── JSON
  └── HTML

Safety and privacy

Agent files are executable Python code. AgentVerdict v0.1.0 does not sandbox them, so only run agent files you trust. See SECURITY.md.

AgentVerdict has no telemetry and does not upload reports. Benchmarks and their outputs remain local unless the agent you supply itself contacts an external service.

Roadmap

See ROADMAP.md for potential post-0.1.0 directions.

Contributing

Contributions are welcome, especially benchmark packs, evaluator types, framework adapters, reporting improvements, and documentation. Read CONTRIBUTING.md before opening a pull request.

License

AgentVerdict is licensed under the MIT License.