ragaeval
A pip-installable Python library providing drop-in RAG (Retrieval-Augmented Generation) evaluation capabilities using the RAGAS framework (version 0.2.15).
Features
- Three Integration Patterns: Choose from decorator, context manager, or manual logging
- Secure Credential Management: Multiple sources with precedence (args > env vars > .env > config file)
- Cross-Platform: Works on Windows, Mac, and Linux
- Automatic Field Normalization: Supports common field name aliases
- CLI Tools: Configuration, execution, and status commands
- Rich Reports: Terminal output with visual indicators and CSV export
Installation
pip install ragaeval
Quick Start
1. Configure API Credentials
Run the interactive configuration wizard:
ragaeval configure
Or set environment variables:
export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o"
export AZURE_OPENAI_API_VERSION="2024-02-15-preview"
2. Log Evaluation Data
Option A: Decorator Pattern
from ragaeval import evaluate_rag
@evaluate_rag
def my_rag_pipeline(query: str, contexts: list) -> str:
# Your RAG logic here
response = generate_answer(query, contexts)
return response
# Use your function normally - data is captured automatically
result = my_rag_pipeline(
query="What is the refund policy?",
contexts=["Returns accepted within 30 days", "Refunds processed in 5-7 days"]
)
Option B: Context Manager Pattern
from ragaeval import EvalSession
with EvalSession() as session:
query = "What is the refund policy?"
contexts = retrieve_documents(query)
response = generate_answer(query, contexts)
session.log(query=query, contexts=contexts, response=response)
Option C: Manual Logging
import ragaeval
query = "What is the refund policy?"
contexts = retrieve_documents(query)
response = generate_answer(query, contexts)
ragaeval.log(query=query, contexts=contexts, response=response)
3. Run Evaluation
ragaeval run
This will:
- Read logged data from
.ragaeval_log.jsonl - Execute RAGAS evaluation using Azure OpenAI
- Display results in terminal with visual indicators
- Export results to
eval_results.csv
CLI Commands
ragaeval run
Execute evaluation on logged data.
ragaeval run # Use defaults
ragaeval run --model gpt-4o # Specify Azure deployment
ragaeval run --output results.csv # Custom output path
ragaeval configure
Interactive credential setup.
ragaeval configure
ragaeval status
Display configuration and log status.
ragaeval status
ragaeval clear
Clear evaluation logs.
ragaeval clear # Prompts for confirmation
ragaeval clear --force # Skip confirmation
ragaeval --version
Display package version.
ragaeval --version
Field Name Aliases
The package supports common field name variations:
- Query:
query,question,input,user_input - Response:
response,answer,llm_response,output,actual_output - Contexts:
contexts,context,retrieved_contexts,source_documents - Reference:
reference,ground_truth,expected
Evaluation Metrics
Always Evaluated (Group A)
- Faithfulness: Response consistency with retrieved contexts
- Response Relevancy: Relevance of response to query
- Aspect Critic: Harmfulness detection
Evaluated When Reference Available (Group B)
- Factual Correctness: Accuracy against ground truth
- Semantic Similarity: Semantic closeness to reference
- BLEU Score: N-gram overlap
- ROUGE Score: Recall-oriented overlap
- String Presence: Exact string matching
- Exact Match: Perfect match detection
Configuration Sources
Credentials are resolved in this order (highest to lowest precedence):
- Explicit function arguments
- Environment variables (
AZURE_OPENAI_API_KEY, etc.) .envfile in current directory- Config file at
~/.ragaeval/config.json
Cross-Platform Notes
- Config file location:
~/.ragaeval/config.json(user home directory) - Log file location:
.ragaeval_log.jsonl(current working directory) - File permissions: Config file is created with user-only access (0o600)
Requirements
- Python >= 3.8
- Azure OpenAI API access (for LLM evaluation)
- Optional: OpenAI API key (for embeddings, otherwise uses local models)
Development
Install in Development Mode
pip install -e .[dev]
Run Tests
pytest
pytest --cov=ragaeval # With coverage
License
MIT License - see LICENSE file for details.
Version
Current version: 0.1.0
Check installed version:
import ragaeval
print(ragaeval.__version__)
Or via CLI:
ragaeval --version
Support
For issues and questions, please file an issue on the GitHub repository.
Metadata
Release files for ragaeval 0.3.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ragaeval-0.3.8.tar.gz | 50.7 kB | Without provenance | Details for ragaeval-0.3.8.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ragaeval-0.3.8-py3-none-any.whl | Python 3 | none | any | Details for ragaeval-0.3.8-py3-none-any.whl |
Total release size: 101.6 kB
Release files / ragaeval-0.3.8.tar.gz
| Download URL | ragaeval-0.3.8.tar.gz |
|---|---|
| Size | 50.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
89f417a43c46933f0f178bb9c093bf560be58a2b936151ce2359d2e533ec0fcc
|
|
BLAKE2b-256 checksum How to use checksums |
513d53602219ae289ac8589ee106446c948a0b38eeebecb7031b8233bca1dbfc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.9
|
Release files / ragaeval-0.3.8-py3-none-any.whl
| Download URL | ragaeval-0.3.8-py3-none-any.whl |
|---|---|
| Size | 50.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cbda6cdfdfcc8bbe01ee3fc2a52e332cfe16351e87bcbdc2d62c9d6c42b32be7
|
|
BLAKE2b-256 checksum How to use checksums |
ea102db7b19aa9af29c6f16480f6063c2a51172579d3639cf2135d011254c9ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.9
|
Security for ragaeval 0.3.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.