Why OpenAgent Eval?

Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.

• Local-first — No cloud services, dashboards, or authentication required
• Framework-agnostic — Works with LangChain, LlamaIndex, or any custom RAG pipeline
• 18+ metrics — Retrieval, generation, faithfulness, relevancy, performance, and cost
• Plugin-based — Extend with custom metrics, providers, and report generators
• Production-ready — Corpus auditing, failure diagnosis, and synthetic test data generation

Installation

pip install openagent-eval

For development:

git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv sync

Quick Start

1. Initialize Configuration

oaeval init --interactive

2. Validate Configuration

oaeval validate config.yaml

3. Run Evaluation

oaeval run config.yaml

4. View Results

oaeval report latest

Features

Evaluation Metrics

Supported Providers

LLM Providers: OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama

Retriever Providers: Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25

CLI Reference

SDK Usage

from openagent_eval.core import Engine
from openagent_eval.config import load_config

config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)

print(report.summary)

Project Structure

openagent-eval/
├── openagent_eval/
│   ├── cli/              # CLI commands (Typer)
│   ├── config/           # Configuration system (Pydantic)
│   ├── core/             # Core orchestration engine
│   ├── metrics/          # 18+ evaluation metrics
│   ├── providers/        # LLM & Retriever adapters
│   ├── corpus/           # Corpus Health Auditor
│   ├── diagnosis/        # Component Diagnosis
│   ├── synthesis/        # Synthetic Test Data
│   ├── reports/          # Report generators
│   └── plugins/          # Plugin system
├── tests/                # Test suite
├── docs/                 # Documentation
└── examples/             # Tutorials and examples

Contributing

We welcome contributions of all kinds. Whether you're fixing a bug, adding a feature, or improving documentation — we'd love your help.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

See CONTRIBUTING.md for detailed guidelines.

Community

• Documentation
• GitHub Issues — Bug reports and feature requests
• GitHub Discussions — Ideas and questions
• Changelog — Release history

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.