rag-sentinel 0.1.10

RAG Evaluation Framework using Ragas metrics and MLflow tracking

RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

1. Create Virtual Environment

# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Package

pip install rag-sentinel

Quick Start

1. Initialize Project

rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - Backend API endpoint and authentication settings
• test_dataset.csv - Sample test dataset

Note: The master configuration (rag_eval_config.yaml) is managed internally by the package. You only need to configure .env and config.ini.

2. Configure

A. Edit .env file

Configure your LLM and Embeddings providers:

# LLM Provider (azure, openai, or ollama)
LLM_PROVIDER=azure

# Azure OpenAI LLM Configuration
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com
AZURE_LLM_DEPLOYMENT_NAME=gpt-4
AZURE_LLM_MODEL=gpt-4
AZURE_LLM_TEMPERATURE=0.0
AZURE_LLM_API_VERSION=2024-02-15-preview

# Embeddings Provider (azure, openai, or ollama)
EMBEDDINGS_PROVIDER=azure

# Azure OpenAI Embeddings Configuration
AZURE_EMBEDDINGS_API_KEY=your-api-key
AZURE_EMBEDDINGS_ENDPOINT=https://your-resource.openai.azure.com
AZURE_EMBEDDINGS_DEPLOYMENT_NAME=text-embedding-ada-002
AZURE_EMBEDDINGS_API_VERSION=2024-02-15-preview

B. Edit config.ini file

Configure your RAG backend API and evaluation settings:

MLflow Settings:

[mlflow]
tracking_uri = http://127.0.0.1:5000
experiment_name = RAG Evaluation
run_name = RAG Evaluation Run

Backend API Endpoint:

[endpoints]
# Your complete RAG API endpoint
response_api_path = https://your-app.com/api/respond

# HTTP method (POST or GET)
method = POST

# Request headers (comma-separated key:value pairs)
headers = Content-Type:application/json

# Request body fields (use {query} and {chat_id} as placeholders)
body_fields = prompt:{query},chat_id:{chat_id}

# JSON response keys
answer_key = response
contexts_key = context

Authentication:

[auth]
# Options: cookie, bearer, header, or none
type = cookie
cookie_name = session
cookie_value = your-session-cookie-value

Dataset:

[dataset]
path = test_dataset.csv
category = simple  # Options: simple (RAGAS) or guardrail (security)

C. Edit test_dataset.csv file

Add your test queries in the format: query,ground_truth,chat_id

Example:

query,ground_truth,chat_id
What is RAG?,RAG stands for Retrieval-Augmented Generation,1
How does vector search work?,Vector search finds similar items using embeddings,2

For detailed configuration help, see the comments in each config file.

3. Validate & Run

# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run

Results will be available in the MLflow UI at the configured tracking URI.

CLI Commands

# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel

Evaluation Categories

Set category in config.ini to choose evaluation type:

Simple (RAGAS Quality Metrics)

category = simple

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

Guardrail (Security Metrics)

category = guardrail

• Toxicity Score - Detects toxic content in responses
• Bias Score - Detects biased content in responses

Performance Metrics

Logged for all evaluation categories:

• Avg Response Time - Average API response time (ms)
• P90 Latency - 90th percentile latency (ms)
• Queries Per Second - Throughput (QPS)

License

MIT