Skip to main content

ragaeval

A pip-installable Python library providing drop-in RAG (Retrieval-Augmented Generation) evaluation capabilities using the RAGAS framework (version 0.2.15).

Features

  • Three Integration Patterns: Choose from decorator, context manager, or manual logging
  • Secure Credential Management: Multiple sources with precedence (args > env vars > .env > config file)
  • Cross-Platform: Works on Windows, Mac, and Linux
  • Automatic Field Normalization: Supports common field name aliases
  • CLI Tools: Configuration, execution, and status commands
  • Rich Reports: Terminal output with visual indicators and CSV export

Installation

pip install ragaeval

Quick Start

1. Configure API Credentials

Run the interactive configuration wizard:

ragaeval configure

Or set environment variables:

export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o"
export AZURE_OPENAI_API_VERSION="2024-02-15-preview"

2. Log Evaluation Data

Option A: Decorator Pattern

from ragaeval import evaluate_rag

@evaluate_rag
def my_rag_pipeline(query: str, contexts: list) -> str:
    # Your RAG logic here
    response = generate_answer(query, contexts)
    return response

# Use your function normally - data is captured automatically
result = my_rag_pipeline(
    query="What is the refund policy?",
    contexts=["Returns accepted within 30 days", "Refunds processed in 5-7 days"]
)

Option B: Context Manager Pattern

from ragaeval import EvalSession

with EvalSession() as session:
    query = "What is the refund policy?"
    contexts = retrieve_documents(query)
    response = generate_answer(query, contexts)
    
    session.log(query=query, contexts=contexts, response=response)

Option C: Manual Logging

import ragaeval

query = "What is the refund policy?"
contexts = retrieve_documents(query)
response = generate_answer(query, contexts)

ragaeval.log(query=query, contexts=contexts, response=response)

3. Run Evaluation

ragaeval run

This will:

  • Read logged data from .ragaeval_log.jsonl
  • Execute RAGAS evaluation using Azure OpenAI
  • Display results in terminal with visual indicators
  • Export results to eval_results.csv

CLI Commands

ragaeval run

Execute evaluation on logged data.

ragaeval run                          # Use defaults
ragaeval run --model gpt-4o          # Specify Azure deployment
ragaeval run --output results.csv    # Custom output path

ragaeval configure

Interactive credential setup.

ragaeval configure

ragaeval status

Display configuration and log status.

ragaeval status

ragaeval clear

Clear evaluation logs.

ragaeval clear               # Prompts for confirmation
ragaeval clear --force       # Skip confirmation

ragaeval --version

Display package version.

ragaeval --version

Field Name Aliases

The package supports common field name variations:

  • Query: query, question, input, user_input
  • Response: response, answer, llm_response, output, actual_output
  • Contexts: contexts, context, retrieved_contexts, source_documents
  • Reference: reference, ground_truth, expected

Evaluation Metrics

Always Evaluated (Group A)

  • Faithfulness: Response consistency with retrieved contexts
  • Response Relevancy: Relevance of response to query
  • Aspect Critic: Harmfulness detection

Evaluated When Reference Available (Group B)

  • Factual Correctness: Accuracy against ground truth
  • Semantic Similarity: Semantic closeness to reference
  • BLEU Score: N-gram overlap
  • ROUGE Score: Recall-oriented overlap
  • String Presence: Exact string matching
  • Exact Match: Perfect match detection

Configuration Sources

Credentials are resolved in this order (highest to lowest precedence):

  1. Explicit function arguments
  2. Environment variables (AZURE_OPENAI_API_KEY, etc.)
  3. .env file in current directory
  4. Config file at ~/.ragaeval/config.json

Cross-Platform Notes

  • Config file location: ~/.ragaeval/config.json (user home directory)
  • Log file location: .ragaeval_log.jsonl (current working directory)
  • File permissions: Config file is created with user-only access (0o600)

Requirements

  • Python >= 3.8
  • Azure OpenAI API access (for LLM evaluation)
  • Optional: OpenAI API key (for embeddings, otherwise uses local models)

Development

Install in Development Mode

pip install -e .[dev]

Run Tests

pytest
pytest --cov=ragaeval  # With coverage

License

MIT License - see LICENSE file for details.

Version

Current version: 0.1.0

Check installed version:

import ragaeval
print(ragaeval.__version__)

Or via CLI:

ragaeval --version

Support

For issues and questions, please file an issue on the GitHub repository.

Metadata

Release files for ragaeval 0.3.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ragaeval 0.3.8
File Size Uploaded Provenance Actions
ragaeval-0.3.8.tar.gz 50.7 kB Without provenance Details for ragaeval-0.3.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ragaeval 0.3.8
File Interpreter ABI Platform Actions
ragaeval-0.3.8-py3-none-any.whl Python 3 none any Details for ragaeval-0.3.8-py3-none-any.whl

Total release size: 101.6 kB

Release files / ragaeval-0.3.8.tar.gz

Download URL ragaeval-0.3.8.tar.gz
Size 50.7 kB
Tags Source
SHA-256 checksum
How to use checksums
89f417a43c46933f0f178bb9c093bf560be58a2b936151ce2359d2e533ec0fcc
BLAKE2b-256 checksum
How to use checksums
513d53602219ae289ac8589ee106446c948a0b38eeebecb7031b8233bca1dbfc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.9

Release files / ragaeval-0.3.8-py3-none-any.whl

Download URL ragaeval-0.3.8-py3-none-any.whl
Size 50.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cbda6cdfdfcc8bbe01ee3fc2a52e332cfe16351e87bcbdc2d62c9d6c42b32be7
BLAKE2b-256 checksum
How to use checksums
ea102db7b19aa9af29c6f16480f6063c2a51172579d3639cf2135d011254c9ad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.9

Security for ragaeval 0.3.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page