trieval

RAG evaluation with exactly 6 metrics. Nothing more.

Every RAG system has 3 variables: Q (Question), C (Context), A (Answer).

3 variables → 6 pairwise relationships → 6 metrics.

The 6 Metrics

Install

pip install trieval

Quick Start

from trieval import Evaluator

evaluator = Evaluator(model="openai:gpt-4o-mini")
result = await evaluator.evaluate(
    question="What is photosynthesis?",
    context="Photosynthesis is the process by which plants convert sunlight into energy.",
    answer="Photosynthesis is how plants make food from sunlight.",
)

print(result.overall_score)       # 0.0–1.0
print(result.retrieval_score)     # avg of C|Q + Q|C
print(result.generation_score)    # avg of A|C + A|Q
print(result.diagnose())          # ["All metrics healthy"] or failure categories

Failure Diagnosis

When your RAG system fails, it's always one of these:

• Retrieval issues — C|Q and Q|C scores are low (wrong context retrieved)
• Generation issues — A|C and A|Q scores are low (bad answer generation)
• End-to-end mismatch — A|C is fine but C|A is low (faithful but unsupported)

Architecture

Built with pydantic-ai (LLM-based metric agents) and LangGraph (evaluation workflow orchestration).

RAGInput(Q, C, A)
    ↓
Evaluator.evaluate()
    ↓
LangGraph: evaluate_metrics → diagnose_failures
    ↓
EvaluationResult (scores + diagnosis)

Each metric is a pydantic-ai Agent with a tailored system prompt. All 6 run concurrently via asyncio.gather inside the LangGraph evaluation node.

Development

uv sync --group dev
pytest                                    # run tests
pytest --cov=trieval --cov-branch         # with coverage
ruff check trieval/ tests/                # lint
ruff format trieval/ tests/               # format
mypy trieval/                             # type check

Documentation

• API Reference — Full API for Evaluator, EvaluationResult, MetricResult, RAGInput, and individual metric functions
• Changelog — Version history

License

MIT