ragaeval

A pip-installable Python library providing drop-in RAG (Retrieval-Augmented Generation) evaluation capabilities using the RAGAS framework (version 0.2.15).

Features

• Three Integration Patterns: Choose from decorator, context manager, or manual logging
• Secure Credential Management: Multiple sources with precedence (args > env vars > .env > config file)
• Cross-Platform: Works on Windows, Mac, and Linux
• Automatic Field Normalization: Supports common field name aliases
• CLI Tools: Configuration, execution, and status commands
• Rich Reports: Terminal output with visual indicators and CSV export

Installation

pip install ragaeval

Quick Start

1. Configure API Credentials

Run the interactive configuration wizard:

ragaeval configure

Or set environment variables:

export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o"
export AZURE_OPENAI_API_VERSION="2024-02-15-preview"

2. Log Evaluation Data

Option A: Decorator Pattern

from ragaeval import evaluate_rag

@evaluate_rag
def my_rag_pipeline(query: str, contexts: list) -> str:
    # Your RAG logic here
    response = generate_answer(query, contexts)
    return response

# Use your function normally - data is captured automatically
result = my_rag_pipeline(
    query="What is the refund policy?",
    contexts=["Returns accepted within 30 days", "Refunds processed in 5-7 days"]
)

Option B: Context Manager Pattern

from ragaeval import EvalSession

with EvalSession() as session:
    query = "What is the refund policy?"
    contexts = retrieve_documents(query)
    response = generate_answer(query, contexts)
    
    session.log(query=query, contexts=contexts, response=response)

Option C: Manual Logging

import ragaeval

query = "What is the refund policy?"
contexts = retrieve_documents(query)
response = generate_answer(query, contexts)

ragaeval.log(query=query, contexts=contexts, response=response)

3. Run Evaluation

ragaeval run

This will:

• Read logged data from .ragaeval_log.jsonl
• Execute RAGAS evaluation using Azure OpenAI
• Display results in terminal with visual indicators
• Export results to eval_results.csv

CLI Commands

ragaeval run

Execute evaluation on logged data.

ragaeval run                          # Use defaults
ragaeval run --model gpt-4o          # Specify Azure deployment
ragaeval run --output results.csv    # Custom output path

ragaeval configure

Interactive credential setup.

ragaeval configure

ragaeval status

Display configuration and log status.

ragaeval status

ragaeval clear

Clear evaluation logs.

ragaeval clear               # Prompts for confirmation
ragaeval clear --force       # Skip confirmation

ragaeval --version

Display package version.

ragaeval --version

Field Name Aliases

The package supports common field name variations:

• Query: query, question, input, user_input
• Response: response, answer, llm_response, output, actual_output
• Contexts: contexts, context, retrieved_contexts, source_documents
• Reference: reference, ground_truth, expected

Evaluation Metrics

Always Evaluated (Group A)

• Faithfulness: Response consistency with retrieved contexts
• Response Relevancy: Relevance of response to query
• Aspect Critic: Harmfulness detection

Evaluated When Reference Available (Group B)

• Factual Correctness: Accuracy against ground truth
• Semantic Similarity: Semantic closeness to reference
• BLEU Score: N-gram overlap
• ROUGE Score: Recall-oriented overlap
• String Presence: Exact string matching
• Exact Match: Perfect match detection

Configuration Sources

Credentials are resolved in this order (highest to lowest precedence):

1. Explicit function arguments
2. Environment variables (AZURE_OPENAI_API_KEY, etc.)
3. .env file in current directory
4. Config file at ~/.ragaeval/config.json

Cross-Platform Notes

• Config file location: ~/.ragaeval/config.json (user home directory)
• Log file location: .ragaeval_log.jsonl (current working directory)
• File permissions: Config file is created with user-only access (0o600)

Requirements

• Python >= 3.8
• Azure OpenAI API access (for LLM evaluation)
• Optional: OpenAI API key (for embeddings, otherwise uses local models)

Development

Install in Development Mode

pip install -e .[dev]

Run Tests

pytest
pytest --cov=ragaeval  # With coverage

License

MIT License - see LICENSE file for details.

Version

Current version: 0.1.0

Check installed version:

import ragaeval
print(ragaeval.__version__)

Or via CLI:

ragaeval --version

Support

For issues and questions, please file an issue on the GitHub repository.