AlignEval

Production-ready Python package for evaluating LLM and RAG systems using GPT-4o/GPT-4.1 as a judge.

AlignEval provides comprehensive evaluation metrics, autonomous agents, synthetic data generation, and multiple integration options (API, CLI, adapters) for seamless incorporation into development workflows. Built on 2025 research best practices including Chain-of-Thought prompting, structured outputs, and position bias mitigation.

Features

🎯 Comprehensive Metrics

• General Metrics: Accuracy, Helpfulness, Relevance, Clarity, Safety
• RAG-Specific Metrics: Faithfulness, Answer Relevancy, Context Precision, Context Recall, Groundedness
• All metrics return 1-5 scores with normalized 0.0-1.0 values and detailed reasoning

🤖 Autonomous Agent

• LangGraph-based agent for end-to-end evaluation workflows
• Automatic metric selection based on test case characteristics
• Actionable recommendations for improving low-scoring responses

🔄 Multiple Integration Options

• Python API: Direct integration with type-safe Pydantic models
• REST API: FastAPI server for microservice architectures
• CLI: Rich terminal interface for quick evaluations
• Adapters: Native support for LangChain, LangGraph, and external REST APIs

📊 Advanced Features

• Batch evaluation with progress tracking
• Async support for high-throughput scenarios
• Synthetic test case generation from domain descriptions or documents
• HTML and JSON report generation
• Position bias mitigation
• Comprehensive error handling with retry logic

Installation

Requirements

• Python 3.10, 3.11, or 3.12
• OpenAI API key with access to GPT-4o or GPT-4.1

Install from PyPI

pip install aligneval

Install from source

git clone https://github.com/aligneval/aligneval.git
cd aligneval
pip install -e .

Development installation

pip install -e ".[dev]"

Quick Start

1. Set up environment variables

Create a .env file in your project root:

ALIGNEVAL_OPENAI_API_KEY=your-openai-api-key-here
ALIGNEVAL_JUDGE_MODEL=gpt-4o  # or gpt-4.1
ALIGNEVAL_TEMPERATURE=0.0
ALIGNEVAL_MAX_TOKENS=1000
ALIGNEVAL_TIMEOUT=60
ALIGNEVAL_MAX_RETRIES=3
ALIGNEVAL_MAX_CONCURRENT_REQUESTS=10
ALIGNEVAL_LOG_LEVEL=INFO
ALIGNEVAL_LOG_FILE=aligneval.log  # Optional: log to file

Or set environment variables directly:

export ALIGNEVAL_OPENAI_API_KEY=your-openai-api-key-here

2. Basic evaluation

from aligneval.core.test_case import LLMTestCase
from aligneval.metrics.accuracy import AccuracyMetric
from aligneval.metrics.helpfulness import HelpfulnessMetric
from aligneval.evaluator.evaluator import AlignEvaluator

# Create a test case
test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris.",
    expected_output="Paris"
)

# Create evaluator with metrics
evaluator = AlignEvaluator(metrics=[
    AccuracyMetric(),
    HelpfulnessMetric()
])

# Evaluate
result = evaluator.evaluate(test_case)

print(f"Overall Score: {result.overall_score:.2f}")
for metric_result in result.metric_results:
    print(f"{metric_result.metric_name}: {metric_result.normalized_score:.2f}")
    print(f"Reasoning: {metric_result.reasoning}")

3. RAG evaluation

from aligneval.core.test_case import LLMTestCase
from aligneval.metrics.faithfulness import FaithfulnessMetric
from aligneval.metrics.answer_relevancy import AnswerRelevancyMetric
from aligneval.evaluator.rag_evaluator import RAGEvaluator

# Create a RAG test case
test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris, which is also its largest city.",
    retrieval_context=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe."
    ]
)

# Create RAG evaluator
evaluator = RAGEvaluator(metrics=[
    FaithfulnessMetric(),
    AnswerRelevancyMetric()
])

# Evaluate with diagnostics
result = evaluator.evaluate(test_case)

print(f"Overall Score: {result.overall_score:.2f}")
print(f"Retrieval Score: {result.diagnostics.retrieval_score:.2f}")
print(f"Generation Score: {result.diagnostics.generation_score:.2f}")
print(f"Bottleneck: {result.diagnostics.bottleneck}")
print(f"Recommendations: {result.diagnostics.recommendations}")

4. Batch evaluation

from aligneval.core.dataset import EvaluationDataset
from aligneval.core.test_case import LLMTestCase
from aligneval.evaluator.evaluator import AlignEvaluator
from aligneval.metrics.accuracy import AccuracyMetric

# Create dataset
dataset = EvaluationDataset(name="my_test_suite")
dataset.add_test_case(LLMTestCase(
    input="What is 2+2?",
    actual_output="4",
    expected_output="4"
))
dataset.add_test_case(LLMTestCase(
    input="What is the capital of Spain?",
    actual_output="Madrid",
    expected_output="Madrid"
))

# Batch evaluate with progress bar
evaluator = AlignEvaluator(metrics=[AccuracyMetric()])
results = evaluator.evaluate_dataset(dataset)

print(f"Evaluated {len(results)} test cases")
passed = sum(1 for r in results if r.passed)
print(f"Passed: {passed}/{len(results)}")

5. Using the CLI

# Evaluate a single test case
aligneval evaluate \
  --input "What is the capital of France?" \
  --output "Paris" \
  --metrics accuracy helpfulness

# Batch evaluate from JSON file
aligneval batch \
  --input dataset.json \
  --metrics accuracy relevance clarity \
  --output results.json

# Generate synthetic test cases
aligneval generate \
  --domain "customer support chatbot" \
  --num-cases 10 \
  --output synthetic_dataset.json

# Generate HTML report
aligneval report \
  --input results.json \
  --output report.html

6. Using the REST API

Start the API server:

# Simple start
uvicorn aligneval.api.main:app --reload

# Or use the convenience script
python run_api.py --reload

# Production with multiple workers
uvicorn aligneval.api.main:app --host 0.0.0.0 --port 8000 --workers 4

The API will be available at:

• API: http://localhost:8000
• Interactive docs: http://localhost:8000/docs
• Health check: http://localhost:8000/api/v1/health

Make requests:

# List available metrics
curl http://localhost:8000/api/v1/metrics

# Evaluate single test case
curl -X POST http://localhost:8000/api/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "test_case": {
      "input": "What is the capital of France?",
      "actual_output": "The capital of France is Paris.",
      "expected_output": "Paris"
    },
    "metrics": ["Accuracy", "Clarity"]
  }'

# Batch evaluation
curl -X POST http://localhost:8000/api/v1/evaluate/batch \
  -H "Content-Type: application/json" \
  -d '{
    "test_cases": [
      {
        "input": "What is 2+2?",
        "actual_output": "4",
        "expected_output": "4"
      },
      {
        "input": "What is the largest planet?",
        "actual_output": "Jupiter",
        "expected_output": "Jupiter"
      }
    ],
    "metrics": ["Accuracy"],
    "max_concurrent": 5
  }'

See aligneval/api/README.md for complete API documentation.

7. Async evaluation for high throughput

import asyncio
from aligneval.core.dataset import EvaluationDataset
from aligneval.evaluator.evaluator import AlignEvaluator
from aligneval.metrics.accuracy import AccuracyMetric

async def evaluate_async():
    # Create dataset
    dataset = EvaluationDataset(name="async_test")
    # ... add test cases ...
    
    # Async batch evaluation with concurrency control
    evaluator = AlignEvaluator(metrics=[AccuracyMetric()])
    results = await evaluator.evaluate_dataset_async(
        dataset,
        max_concurrent=5  # Control API rate limits
    )
    
    return results

# Run async evaluation
results = asyncio.run(evaluate_async())

Configuration

AlignEval uses environment variables for configuration. All variables are prefixed with ALIGNEVAL_:

Variable	Default	Description
ALIGNEVAL_OPENAI_API_KEY	required	OpenAI API key
ALIGNEVAL_JUDGE_MODEL	gpt-4o	Judge model (gpt-4o or gpt-4.1)
ALIGNEVAL_TEMPERATURE	0.0	Temperature for judge LLM (0.0-2.0)
ALIGNEVAL_MAX_TOKENS	1000	Maximum tokens for responses
ALIGNEVAL_TIMEOUT	60	API call timeout in seconds
ALIGNEVAL_MAX_RETRIES	3	Maximum retry attempts for failed calls
ALIGNEVAL_RETRY_DELAY	1.0	Initial retry delay (uses exponential backoff)
ALIGNEVAL_MAX_CONCURRENT_REQUESTS	10	Max concurrent requests for async batch evaluation
ALIGNEVAL_ENABLE_POSITION_BIAS_MITIGATION	true	Enable output randomization
ALIGNEVAL_LOG_LEVEL	INFO	Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
ALIGNEVAL_LOG_FILE	None	Optional log file path

⚡ Performance Optimization

AlignEval uses OpenAI's GPT models as judge LLMs, which introduces API latency. Here's how to optimize performance:

Speed Comparison

Method	100 Test Cases (5 metrics)	Speed
Sync (sequential)	~16 minutes	1x baseline
Async (10 concurrent)	~1.7 minutes	10x faster ⚡
Async (20 concurrent)	~50 seconds	20x faster ⚡⚡
Async + gpt-4o-mini	~25 seconds	40x faster ⚡⚡⚡

Quick Wins

1. Use Async for Batch Evaluation (10x faster!)

import asyncio
from aligneval import AlignEvaluator, EvaluationDataset
from aligneval.metrics import AccuracyMetric, RelevanceMetric

evaluator = AlignEvaluator([AccuracyMetric(), RelevanceMetric()])
dataset = EvaluationDataset.from_json("test_cases.json")

# Async with concurrency control
async def main():
    results = await evaluator.evaluate_dataset_async(
        dataset,
        max_concurrent=20  # Adjust based on rate limits
    )

asyncio.run(main())

2. Use Fewer Metrics (5x faster)

# ❌ Slow: 10 metrics = 10 API calls per test case
evaluator = AlignEvaluator(get_all_metrics())

# ✅ Fast: 2 metrics = 2 API calls per test case
evaluator = AlignEvaluator([AccuracyMetric(), RelevanceMetric()])

3. Use Faster Model (2-3x faster)

# Use gpt-4o-mini instead of gpt-4o
export ALIGNEVAL_JUDGE_MODEL="gpt-4o-mini"

4. Increase Concurrency (2-5x faster)

# Default: 10 concurrent requests
export ALIGNEVAL_MAX_CONCURRENT_REQUESTS=10

# Faster: 20-50 concurrent (check your OpenAI rate limits!)
export ALIGNEVAL_MAX_CONCURRENT_REQUESTS=20

Performance Demo

Run the performance comparison:

python examples/performance_demo.py

For detailed optimization strategies, see PERFORMANCE_OPTIMIZATION_GUIDE.md.

Available Metrics

General Metrics

• AccuracyMetric: Evaluates factual correctness
• HelpfulnessMetric: Evaluates utility and usefulness
• RelevanceMetric: Evaluates pertinence to the query
• ClarityMetric: Evaluates understandability and coherence
• SafetyMetric: Evaluates harmlessness and absence of toxic content

RAG-Specific Metrics

• FaithfulnessMetric: Evaluates if response is grounded in context
• AnswerRelevancyMetric: Evaluates if response addresses the query
• ContextPrecisionMetric: Evaluates if retrieved context is relevant
• ContextRecallMetric: Evaluates if all necessary information was retrieved
• GroundednessMetric: Evaluates factual consistency with context

All metrics return:

• raw_score: Integer from 1-5
• normalized_score: Float from 0.0-1.0 (calculated as (raw_score - 1) / 4.0)
• reasoning: Chain-of-thought explanation from the judge LLM
• metadata: Additional information about the evaluation

Examples

See the examples/ directory for complete working examples:

• evaluator_demo.py: Basic single and batch evaluation
• rag_evaluator_demo.py: RAG system evaluation with diagnostics
• async_evaluation.py: High-throughput async evaluation
• custom_metrics.py: Creating custom metrics
• report_generation.py: Generating HTML and JSON reports

Documentation

• Full Documentation
• API Reference
• Examples
• Contributing Guide

Architecture

AlignEval follows a layered architecture:

┌─────────────────────────────────────────────────────────┐
│                    Integration Layer                     │
│  (FastAPI API, Click CLI, LangChain/LangGraph Adapters) │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                   Application Layer                      │
│  (AlignEvaluator, RAGEvaluator, Pipeline, Agent)       │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                      Metrics Layer                       │
│     (BaseMetric, 5 General Metrics, 5 RAG Metrics)     │
└─────────────────────────────────────────────────────────┘
                            │
┌─────────────────────────────────────────────────────────┐
│                       Core Layer                         │
│    (LLMTestCase, EvaluationDataset, Settings, Config)  │
└─────────────────────────────────────────────────────────┘

Technology Stack

• Core: Python 3.10+, Pydantic 2.5+
• LLM: OpenAI GPT-4o/GPT-4.1
• Orchestration: LangChain 0.3.x, LangGraph 0.2.x
• API: FastAPI 0.115.x, Uvicorn
• CLI: Click 8.x, Rich 13.x
• Testing: pytest 7.x, pytest-asyncio, hypothesis

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

MIT License - see LICENSE file for details.

Citation

If you use AlignEval in your research, please cite:

@software{aligneval2025,
  title = {AlignEval: Production-ready LLM and RAG Evaluation Framework},
  author = {AlignEval Team},
  year = {2025},
  url = {https://github.com/aligneval/aligneval}
}

Support

• GitHub Issues
• Documentation
• Discord Community

Acknowledgments

Built on research best practices from:

• MT-Bench: Multi-turn benchmark for LLM evaluation
• G-Eval: LLM-based evaluation with Chain-of-Thought
• RAGAS: RAG assessment framework