rag-sentinel 0.1.6

RAG Evaluation Framework using Ragas metrics and MLflow tracking

RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

1. Create Virtual Environment

# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Package

pip install rag-sentinel

Quick Start

1. Initialize Project

rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - App settings and authentication
• rag_eval_config.yaml - Master configuration
• test_dataset.csv - Sample test dataset

2. Configure

Edit .env:

AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-02-15-preview
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-4
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-ada-002

Edit config.ini:

[app]
app_url = https://your-rag-app.com/backend

[endpoints]
context_api_path = /api/retrieve_context
answer_api_path = /api/respond

[auth]
type = cookie
cookie_name = session
cookie_value = your-session-cookie

[mlflow]
tracking_uri = http://127.0.0.1:5001

3. Edit Test Dataset

Edit test_dataset.csv:

chat_id,query,ground_truth
1,What is RAG?,RAG stands for Retrieval-Augmented Generation
2,How does it work?,RAG combines retrieval and generation

4. Validate & Run

# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run

Results will be available in the MLflow UI at the configured tracking URI.

CLI Commands

# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel

Metrics

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

License

MIT