RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

1. Create Virtual Environment

# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Package

pip install rag-sentinel

Quick Start

1. Initialize Project

rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - Backend API endpoint and authentication settings
• test_dataset.csv - Sample test dataset

Note: The master configuration (rag_eval_config.yaml) is managed internally by the package. You only need to configure .env and config.ini.

2. Configure

A. Edit .env file

Configure your LLM and Embeddings providers:

Supported LLM Providers: azure, openai, ollama, gemini Supported Embeddings Providers: azure, openai, ollama, huggingface

Example 1: Azure OpenAI

# LLM Provider
LLM_PROVIDER=azure

# Azure OpenAI LLM Configuration
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com
AZURE_LLM_DEPLOYMENT_NAME=gpt-4
AZURE_LLM_MODEL=gpt-4
AZURE_LLM_TEMPERATURE=0.0
AZURE_LLM_API_VERSION=2024-02-15-preview

# Embeddings Provider
EMBEDDINGS_PROVIDER=azure

# Azure OpenAI Embeddings Configuration
AZURE_EMBEDDINGS_API_KEY=your-api-key
AZURE_EMBEDDINGS_ENDPOINT=https://your-resource.openai.azure.com
AZURE_EMBEDDINGS_DEPLOYMENT_NAME=text-embedding-ada-002
AZURE_EMBEDDINGS_API_VERSION=2024-02-15-preview

Example 2: Google Gemini + HuggingFace Embeddings

# LLM Provider
LLM_PROVIDER=gemini

# Google Gemini LLM Configuration
GEMINI_API_KEY=your-google-api-key
GEMINI_MODEL=gemini-2.0-flash-exp
GEMINI_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=huggingface

# HuggingFace Embeddings Configuration
HUGGINGFACE_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2
HUGGINGFACE_API_KEY=  # Optional: leave empty for local models

Example 3: OpenAI

# LLM Provider
LLM_PROVIDER=openai

# OpenAI LLM Configuration
OPENAI_LLM_API_KEY=your-openai-api-key
OPENAI_LLM_MODEL=gpt-4
OPENAI_LLM_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=openai

# OpenAI Embeddings Configuration
OPENAI_EMBEDDINGS_API_KEY=your-openai-api-key
OPENAI_EMBEDDINGS_MODEL=text-embedding-3-small

Example 4: Ollama (Local)

# LLM Provider
LLM_PROVIDER=ollama

# Ollama LLM Configuration
OLLAMA_LLM_BASE_URL=http://localhost:11434
OLLAMA_LLM_MODEL=llama2
OLLAMA_LLM_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=ollama

# Ollama Embeddings Configuration
OLLAMA_EMBEDDINGS_BASE_URL=http://localhost:11434
OLLAMA_EMBEDDINGS_MODEL=nomic-embed-text

B. Edit config.ini file

Configure your RAG backend API and evaluation settings:

MLflow Settings:

[mlflow]
tracking_uri = http://127.0.0.1:5000
experiment_name = RAG Evaluation
run_name = RAG Evaluation Run

Backend API Endpoint:

[endpoints]
# Your complete RAG API endpoint
response_api_path = https://your-app.com/api/respond

# HTTP method (POST or GET)
method = POST

# Request headers (comma-separated key:value pairs)
headers = Content-Type:application/json

# Request body fields (use {query} and {chat_id} as placeholders)
body_fields = prompt:{query},chat_id:{chat_id}

# JSON response keys
answer_key = response
contexts_key = context

Authentication:

[auth]
# Options: cookie, bearer, header, or none
type = cookie
cookie_name = session
cookie_value = your-session-cookie-value

Dataset:

[dataset]
path = test_dataset.csv
category = simple  # Options: simple (RAGAS) or guardrail (security)

C. Edit test_dataset.csv file

Add your test queries in the format: query,ground_truth,chat_id

Example:

query,ground_truth,chat_id
What is RAG?,RAG stands for Retrieval-Augmented Generation,1
How does vector search work?,Vector search finds similar items using embeddings,2

For detailed configuration help, see the comments in each config file.

3. Validate & Run

# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run

Results will be available in the MLflow UI at the configured tracking URI.

CLI Commands

# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel

Evaluation Categories

Set category in config.ini to choose evaluation type:

Simple (RAGAS Quality Metrics)

category = simple

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

Guardrail (Security Metrics)

category = guardrail

• Toxicity Score - Detects toxic content in responses
• Bias Score - Detects biased content in responses

Performance Metrics

Logged for all evaluation categories:

• Avg Response Time - Average API response time (ms)
• P90 Latency - 90th percentile latency (ms)
• Queries Per Second - Throughput (QPS)

Supported Providers

LLM Providers

Embeddings Providers

Changelog

Version 0.1.11 (2026-03-25)

New Features:

• ✨ Added support for Google Gemini LLM provider
• ✨ Added support for HuggingFace embeddings (including MiniLM-L6-v2)
• 📚 Enhanced documentation with provider examples

Dependencies:

• Added langchain-google-genai>=1.0.0
• Added langchain-huggingface>=0.0.1

Version 0.1.10

• Initial stable release with Azure, OpenAI, and Ollama support

License

MIT