agent-eval

Framework for evaluating AI agents across multiple quality dimensions

Part of the AumOS open-source agent infrastructure portfolio.

Features

• EvalRunner orchestrates multi-run evaluation with configurable concurrency, per-case timeouts, retry logic, and fail-fast mode
• Six evaluation dimensions — ACCURACY, LATENCY, COST, SAFETY, FORMAT, and CUSTOM — each producing a normalized [0.0, 1.0] score with a pass/fail determination
• BenchmarkSuite YAML loader with per-case expected outputs, latency budgets, and cost caps; SuiteBuilder for programmatic construction
• LLM-judge evaluator alongside deterministic accuracy, latency, cost, and format evaluators — all implementing the Evaluator ABC
• Agent adapters for LangChain, CrewAI, AutoGen, OpenAI Agents, and plain callables so any agent can be wrapped without code changes
• Reporting in JSON, Markdown, HTML, and rich console output with per-dimension aggregate statistics
• Quality gates (ThresholdGate, CompositeGate) that turn evaluation results into CI pass/fail signals

Quick Start

Install from PyPI:

pip install agent-eval

Verify the installation:

agent-eval version

Basic usage:

import agent_eval

# See examples/01_quickstart.py for a working example

Documentation

• Architecture
• Contributing
• Changelog
• Examples

Enterprise Upgrade

For production deployments requiring SLA-backed support and advanced integrations, contact the maintainers or see the commercial extensions documentation.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request.

License

Apache 2.0 — see LICENSE for full terms.

Part of AumOS — open-source agent infrastructure.