Skip to main content

AI Agent Evaluation

CI License: MIT Python 3.10+

Should your agent switch models? Find out in an afternoon, not a day.

Run your agent's test cases on the model you use today and the one you are considering. You get one honest answer, pass, regressed or not enough data, with the cost and speed of each model next to it.

A shopping assistant compared on two models: test cases in, one answer out

The problem

A newer or cheaper model comes out, or the old one is retired. Before switching, someone has to rerun every test case on the new model and compare the results by hand.

  • It is slow. A day of reruns and side by side reading for one decision.
  • One run per case proves little. Agents answer differently every time. In our simulations, comparing one run per case flagged a regression in 48% to 100% of swaps where nothing had changed.
  • Nobody can say what "about the same" means. Does it include the payments flow? Is the new model 3 points worse or 15?
  • Cost and speed sit somewhere else. They are usually the reason to switch, but they are not on the same page as quality.

Who it is for

  • Developers on small teams whose agent already has test cases, or a script that runs it a few times.
  • Agents whose results code can check: the right product was recommended, the right tool was called, the output parses. Answers that need a person or a model to grade them are not supported yet.

What it does

  • Runs both models on your cases, 6 runs per case per model, shuffled so a slow minute at the provider hits both alike.
  • Gives one answer for the suite: pass if the new model is at most your margin worse (10 points by default), regressed if it is clearly worse, not enough data if the runs cannot tell. (non-inferiority test)
  • Stops early when the answer is already clear, after half the runs. (two looks)
  • Never lets a critical case hide in the average. Mark it must pass and it can fail the comparison on its own.
  • Keeps provider outages out of the score. Rate limits and server errors are counted separately, not blamed on the model.
  • Shows what each correct answer really costs. Cost per correct task counts every run, failed ones and extra model calls included, so a model that is cheaper per token but fails more cannot look like a saving. Speed, model calls and reasoning tokens sit next to it.
  • Works where you test: a command, a pytest helper, and a GitHub workflow you can copy.

Example: a shopping assistant

A shopping assistant reads a request, searches a product catalog with a search_products tool, and either recommends one product, asks a question, or says nothing fits. It is written for the OpenAI Python SDK, with tool calling. To run without an API key, it also has simulated models (large-model, small-model, tiny-model) with different accuracy.

1. The test cases are a JSONL file, one shopper request per line with the answer it should get (examples/shopping_assistant/cases.jsonl):

{"name": "running shoes under $100 size 10", "input": "Find running shoes under $100 in size 10", "expected": {"action": "recommend", "product_id": "RUN-101"}, "must_pass": true}
{"name": "running shoes no size", "input": "Find running shoes under $120", "expected": {"action": "ask"}}
{"name": "headphones under $40", "input": "Any headphones under $40?", "expected": {"action": "none"}}

2. The suite points at the agent and the file (examples/shopping_assistant/suite.py):

from agent import shopping_agent          # shopping_agent(request, model) returns {"action", "product_id", "message"}
from ai_agent_evaluation import Suite

suite = Suite.from_file(
    "cases.jsonl",
    agent=shopping_agent,
    prices={"large-model": (2.00, 8.00), "small-model": (0.40, 1.60)},   # USD per million tokens
)

3. Run it:

git clone https://github.com/dhruv1999/AI-Agent-Evaluation && cd AI-Agent-Evaluation
pip install -e .
ai-agent-evaluation compare examples/shopping_assistant/suite.py --baseline-model large-model --candidate-model small-model

4. The output:

Model swap: large-model -> small-model   (suite shopping_assistant, 30 cases)

PASS. The new model is at most 2.6 points worse, inside your 10 point margin.
Changes up to 10 points worse are accepted; change this with --margin.

                    large-model   small-model
pass rate                 93.3%         96.7%
cost per run            $0.0045       $0.0011   77% lower
cost per correct        $0.0049       $0.0011   77% lower
median time               0.02s         0.01s   49% faster
calls per run               2.0           2.3   12% more
reasoning tokens            300           120

Must pass:
  running shoes under $100 size 10: 4/6 -> 5/6  ok
  headphones under $100: 6/6 -> 6/6  ok
  To fail on any single failure instead, mark the case strict=True.

Runs: 360 of a 360 budget, stopped at look 2 of 2. Seed 7.

Compare with tiny-model, which often ignores the shopper's budget, and the answer changes:

REGRESSED. The new model is at least 12.1 points worse, beyond your 10 point margin.

                    large-model    tiny-model
pass rate                 91.1%         62.2%
cost per run            $0.0046       $0.0003   93% lower
cost per correct        $0.0050       $0.0005   90% lower
median time               0.02s         0.01s   68% faster
calls per run               2.1           2.6   25% more
reasoning tokens            300             -

It stopped after half the runs, because the answer was already clear. Each comparison also writes eval-results/eval-report.md and eval-results/eval-verdict.json.

With real OpenAI models: set OPENAI_API_KEY and SHOP_LIVE=1, pass real model names, and give their prices with --price MODEL=INPUT,OUTPUT. The output above comes from the simulated models (with DEMO_SEED=7 ... --seed 7), not from real ones.

To use the library in your own project:

pip install git+https://github.com/dhruv1999/AI-Agent-Evaluation

Test case files

Cases come from JSONL (one JSON object per line) or CSV (one row per case, editable in Excel or Google Sheets).

Field Required Meaning
input yes What the agent receives: text, or any JSON value
expected usually What a correct answer contains. expected_output and target also work, so datasets from other eval tools load as they are
name no A unique name; defaults to "case 1", "case 2"
must_pass no true lets this case fail the comparison on its own
strict no true fails the comparison on any single failed run of this case
check no Which check to use for this case
name,input,expected,must_pass
budget running shoes,"Find running shoes under $100 in size 10","{""action"": ""recommend"", ""product_id"": ""RUN-101""}",true
shoes without a size,Find running shoes under $120,"{""action"": ""ask""}",false

Checks. By default each run is compared with expected:

  • matches_expected (default): for an object, every expected field must match and other fields are ignored; for a list, the answer must be one of its items; otherwise the answer must equal it.
  • equals_expected: the answer must equal expected exactly.
  • contains_expected: the answer, as text, must contain expected.

For anything else, write a check in Python and pass it: Suite.from_file("cases.jsonl", agent=agent, check=my_check), or name it per case with checks={"my_check": my_check}. A check returns False, or raises AssertionError, to fail a run.

Prefer code? Cases can also be written directly in Python with Case(name, input, check=...); see examples/xpath_healing.

Start with --smoke for a single quick run per case. It checks your setup and is labelled as not a verdict.

Results and exit codes

Answer Meaning Exit code
pass The new model is at most your margin worse 0
regressed Clearly worse than your margin, or a must pass case clearly dropped 1
not enough data The runs could not tell. The report suggests how many runs would 10, or 0 with --on-inconclusive warn
errors More than 20% of runs failed at the provider, so there is no verdict 11

In pytest:

from ai_agent_evaluation import assert_model_swap
from agent_suite import suite

def test_switch_to_cheaper_model():
    assert_model_swap(suite, "gpt-current", "gpt-cheaper")

How well it works

From 2,000 simulated model swaps per row, with the defaults (6 runs per case per model, 10 point margin), on a 40 case suite with a mix of stable and flaky cases. Full tables: docs/how-it-decides.md.

The new model really is This library Running each case once
The same passes 99.0%, never says regressed says regressed 91.6% of the time
10 points worse (exactly the margin) passes only 2.1% says regressed 99.9%
20 points worse says regressed 88.3%, never passes says regressed 100%
A must pass case breaks, the rest the same fails it 100% flags it 100%, along with most unchanged swaps

The honest limits:

  • Small suites need more runs. With 10 cases, 6 runs settle only about half of unchanged swaps. Use --runs 16.
  • Very flaky agents need more runs too. Where cases pass about 70% of the time, 27% of unchanged swaps end as not enough data.
  • Judgement calls are out of scope. Agents whose answers need an LLM judge are not supported in this version.

Roadmap

  • Run on every pull request, with a comment on the pull request and a cached baseline.
  • Retries that wait out rate limits, and stopping early during an outage.
  • Answers graded by an LLM judge, with a check for judge drift.
  • Reports across more than two models.

Docs

MIT licensed.

Metadata

Release files for ai-agent-evaluation 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ai-agent-evaluation 0.2.0
File Size Uploaded Provenance Actions
ai_agent_evaluation-0.2.0.tar.gz 321.7 kB With provenance Details for ai_agent_evaluation-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ai-agent-evaluation 0.2.0
File Interpreter ABI Platform Actions
ai_agent_evaluation-0.2.0-py3-none-any.whl Python 3 none any Details for ai_agent_evaluation-0.2.0-py3-none-any.whl

Total release size: 351.0 kB

Release files / ai_agent_evaluation-0.2.0.tar.gz

Download URL ai_agent_evaluation-0.2.0.tar.gz
Size 321.7 kB
Tags Source
SHA-256 checksum
How to use checksums
f8ff2325408a940295b099dca436f1ae54f7e8655cc7fbf38d840daeae8845a8
BLAKE2b-256 checksum
How to use checksums
cad49e723ceb13c1b14de94d450a8267bad1d3922f43198fa3ea8987ff450863
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / ai_agent_evaluation-0.2.0-py3-none-any.whl

Download URL ai_agent_evaluation-0.2.0-py3-none-any.whl
Size 29.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7dd7b747dceae30de3af547dedb08ca923a774166e734edf2c70b9a51d9c7b0f
BLAKE2b-256 checksum
How to use checksums
85050aaa9cfb986a864341c7ecd87611724660d3a21ace4066d72e49cd19e76b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for ai-agent-evaluation 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ai_agent_evaluation-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page