acp-evals 0.1.0

Comprehensive evaluation framework for Agent Communication Protocol (ACP) agents

ACP Evals

Production-ready evaluation framework for agents in the ACP/BeeAI ecosystem

Overview

ACP Evals provides comprehensive evaluation tools for agents built with the Agent Communication Protocol (ACP). It enables developers to measure, benchmark, and improve agent performance with a focus on multi-agent systems, production metrics, and developer experience.

Key Features

• 🚀 Zero to evaluation in < 5 lines of code
• 🤖 Multi-agent focused - Specialized metrics for agent coordination
• 🔌 Multiple LLM providers - OpenAI, Anthropic, Ollama, or mock mode
• 📊 Production metrics - Token usage, costs, latency tracking
• 🎯 Built-in evaluators - Accuracy, performance, reliability, safety

Quick Start

from acp_evals import evaluate, AccuracyEval

# Evaluate any agent with just 3 lines
result = evaluate(
    AccuracyEval(agent="http://localhost:8000/agents/my-agent"),
    input="What is the capital of France?",
    expected="Paris"
)

Documentation

• 📚 Full Documentation
• 🚀 Getting Started
• 🏗️ Architecture Guide
• 🔧 Provider Setup
• 💡 Examples

Installation

pip install acp-evals

# Or with specific provider support
pip install "acp-evals[openai]"
pip install "acp-evals[anthropic]"
pip install "acp-evals[all-providers]"

Project Structure

acp-evals/
├── python/                 # Python implementation
│   ├── src/acp_evals/     # Core package
│   ├── tests/             # Test suite
│   ├── examples/          # Example scripts
│   └── docs/              # Documentation
└── internal-docs/         # Internal planning documents

Contributing

We welcome contributions! Please see the Python contributing guide for details.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Community

• 💬 GitHub Discussions
• 🐛 Issue Tracker
• 💬 Discord Community

---

Part of the BeeAI project, an initiative of the Linux Foundation AI & Data