aligneval 0.1.0

An open-source Python framework for evaluating LLM systems.

AlignEval

AlignEval is an open-source Python framework for evaluating LLM systems. It helps you measure accuracy, hallucination, relevancy, and safety across your RAG pipelines, agents, and custom chatbots.

Features

• Async Evaluation: Evaluate multiple test cases concurrently for speed.
• Pluggable Metrics: Use built-in metrics (Accuracy, Hallucination, Safety) or define your own.
• Caching: Avoid re-running expensive evaluations with built-in caching.
• Synthetic Data: Generate test cases using an LLM to bootstrap your evaluation dataset.
• Pytest Integration: Run evaluations as part of your standard test suite.

Installation

pip install aligneval

Quick Start

import asyncio
from aligneval import Evaluator
from aligneval.metrics import Accuracy, Hallucination

async def main():
    evaluator = Evaluator()
    dataset = [
        {"input": "What is the capital of France?", "output": "Paris", "context": "France is a country in Europe. Its capital is Paris.", "expected": "Paris"},
        # ... more cases
    ]
    
    results = await evaluator.evaluate(
        dataset,
        metrics=[Accuracy(), Hallucination()]
    )
    
    print(results)

if __name__ == "__main__":
    asyncio.run(main())