ACP Evals

ACP Evals is an open framework for evaluating AI agents across accuracy, performance, and reliability dimensions.

Modern AI agents need comprehensive testing before deployment. ACP Evals provides production-grade evaluation using LLM-as-judge methodology, designed to integrate seamlessly with the BeeAI ecosystem and any ACP-compliant agent.

ACP Evals enables you to:

• Measure response accuracy using configurable LLM judges
• Track performance metrics including latency and memory usage
• Validate tool usage patterns and error handling
• Run batch evaluations for comprehensive test coverage
• Generate detailed reports for continuous improvement

Core Concepts

Quick Example

Evaluate agent accuracy with just a few lines:

from acp_evals import AccuracyEval

evaluation = AccuracyEval(
    agent="http://localhost:8001/agents/my-agent",
    rubric="factual"
)

result = await evaluation.run(
    input="What is 10*5 then to the power of 2? do it step by step",
    expected="2500",
    print_results=True
)
assert result is not None and result.score >= 0.7

Core Features

• Comprehensive Evaluation - Run all three evaluation dimensions in a single command
• Rich TUI Display - Interactive terminal UI with detailed metrics and LLM judge explanations
• Batch Testing - Evaluate multiple test cases with parallel execution
• Multiple Provider Support - Works with OpenAI, Anthropic, Ollama, and more
• Export Capabilities - Generate JSON reports for CI/CD integration

Installation

pip install acp-evals

Quickstart

1. Configure your LLM provider

echo "OPENAI_API_KEY=your-key-here" > .env
acp-evals check

2. Run your first evaluation

# Test accuracy
acp-evals run accuracy http://localhost:8001/agents/my-agent \
  -i "What is 2+2?" -e "4"

3. Run comprehensive evaluation

acp-evals comprehensive http://localhost:8001/agents/my-agent \
  -i "Calculate compound interest" -e "Detailed calculation"

Examples

Performance Evaluation

from acp_evals import PerformanceEval

evaluation = PerformanceEval(
    agent="http://localhost:8001/agents/my-agent",
    num_iterations=5,
    track_memory=True
)

result = await evaluation.run(
    input_text="What is the capital of France?",
    print_results=True
)

Reliability Evaluation

from acp_evals import ReliabilityEval

evaluation = ReliabilityEval(
    agent="http://localhost:8001/agents/my-agent",
    tool_definitions=["search", "calculator"]
)

result = await evaluation.run(
    input="Search for AAPL price and calculate P/E ratio",
    expected_tools=["search", "calculator"],
    print_results=True
)
assert result.passed

Agent Formats

ACP Evals works with any agent implementation:

• ACP-compliant agents: http://localhost:8001/agents/my-agent
• Python functions: agent.py:function_name
• Python modules: mymodule.agent_function

CLI Reference

# Check setup
acp-evals check

# Run evaluations
acp-evals run accuracy <agent> -i <input> -e <expected>
acp-evals run performance <agent> -i <input>
acp-evals run reliability <agent> -i <input> --expected-tools <tool>

# Comprehensive testing
acp-evals comprehensive <agent> -i <input> -e <expected>

# Batch testing
acp-evals run accuracy <agent> --test-file tests.jsonl

Resources

• Documentation - API reference and guides
• Examples - Ready-to-run code samples
• Issues - Report bugs or request features

License

Apache 2.0 - see LICENSE

Developed by contributors to the BeeAI project, this initiative is part of the Linux Foundation AI & Data program. Its development follows open, collaborative, and community-driven practices.