openagent-eval 0.4.10

Why OpenAgent Eval?

Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.

• Local-first — No cloud services, dashboards, or authentication required
• Framework-agnostic — Works with LangChain, LlamaIndex, or any custom RAG pipeline
• 18+ metrics — Retrieval, generation, faithfulness, relevancy, performance, and cost
• Plugin-based — Extend with custom metrics, providers, and report generators
• Production-ready — Corpus auditing, failure diagnosis, and synthetic test data generation

Installation

pip install openagent-eval

For development:

git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv sync

Quick Start

1. Initialize Configuration

oaeval init --interactive

The wizard walks you through your dataset path, LLM provider and model, retriever, metric preset, and report format, then writes a ready-to-run config.yaml.

2. Validate Configuration

oaeval validate config.yaml

3. Run Evaluation

oaeval run config.yaml

4. View Results

oaeval report latest

Contributing & Community

OpenAgent Eval is community-built and welcomes contributors of every experience level. If you've used the tool, you already know something worth sharing — a bug, a rough edge in the docs, or a metric you wish existed.

Ways to help:

• Code — pick up a good first issue or something labeled help wanted, fix a bug, or add a metric or provider.
• Docs — clarify a guide, fix a typo, or expand the documentation.
• Bug reports — hit something unexpected? Open an issue with steps to reproduce.
• Tutorials & examples — show how you use OpenAgent Eval by adding to examples/.

Getting started:

1. Fork the repository and create a feature branch (git checkout -b feature/your-change)
2. Make your change and add tests where it makes sense
3. Run the checks (uv run pytest and uv run ruff check .)
4. Open a pull request

New to the project? CONTRIBUTING.md walks through setup, the development workflow, and coding conventions in detail.

Where to connect:

• GitHub Discussions — ask questions, share ideas, and show what you built
• GitHub Issues — bug reports and feature requests
• Documentation — guides and API reference
• Changelog — release history

Features

Evaluation Metrics

Supported Providers

LLM Providers: OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama

Retriever Providers: Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25

CLI Reference

SDK Usage

from openagent_eval.core import Engine
from openagent_eval.config import load_config

config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)

print(report.summary)

Project Structure

openagent-eval/
├── openagent_eval/
│   ├── cli/              # CLI commands (Typer)
│   ├── config/           # Configuration system (Pydantic)
│   ├── core/             # Core orchestration engine
│   ├── metrics/          # 18+ evaluation metrics
│   ├── providers/        # LLM & Retriever adapters
│   ├── corpus/           # Corpus Health Auditor
│   ├── diagnosis/        # Component Diagnosis
│   ├── synthesis/        # Synthetic Test Data
│   ├── reports/          # Report generators
│   └── plugins/          # Plugin system
├── tests/                # Test suite
├── docs/                 # Documentation
└── examples/             # Tutorials and examples

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.