rag-sentinel 0.1.12

RAG Evaluation Framework using Ragas metrics and MLflow tracking

RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

1. Create Virtual Environment

# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Package

pip install rag-sentinel

Quick Start

1. Initialize Project

rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - Backend API endpoint and authentication settings
• test_dataset.csv - Sample test dataset

Note: The master configuration (rag_eval_config.yaml) is managed internally by the package. You only need to configure .env and config.ini.

2. Configure

A. Edit .env file

Configure your LLM and Embeddings providers:

Supported LLM Providers: azure, openai, ollama, gemini Supported Embeddings Providers: azure, openai, ollama, huggingface

Example 1: Azure OpenAI

# LLM Provider
LLM_PROVIDER=azure

# Azure OpenAI LLM Configuration
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com
AZURE_LLM_DEPLOYMENT_NAME=gpt-4
AZURE_LLM_MODEL=gpt-4
AZURE_LLM_TEMPERATURE=0.0
AZURE_LLM_API_VERSION=2024-02-15-preview

# Embeddings Provider
EMBEDDINGS_PROVIDER=azure

# Azure OpenAI Embeddings Configuration
AZURE_EMBEDDINGS_API_KEY=your-api-key
AZURE_EMBEDDINGS_ENDPOINT=https://your-resource.openai.azure.com
AZURE_EMBEDDINGS_DEPLOYMENT_NAME=text-embedding-ada-002
AZURE_EMBEDDINGS_API_VERSION=2024-02-15-preview

Example 2: Google Gemini + HuggingFace Embeddings

# LLM Provider
LLM_PROVIDER=gemini

# Google Gemini LLM Configuration
GEMINI_API_KEY=your-google-api-key
GEMINI_MODEL=gemini-2.0-flash-exp
GEMINI_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=huggingface

# HuggingFace Embeddings Configuration
HUGGINGFACE_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2
HUGGINGFACE_API_KEY=  # Optional: leave empty for local models

Example 3: OpenAI

# LLM Provider
LLM_PROVIDER=openai

# OpenAI LLM Configuration
OPENAI_LLM_API_KEY=your-openai-api-key
OPENAI_LLM_MODEL=gpt-4
OPENAI_LLM_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=openai

# OpenAI Embeddings Configuration
OPENAI_EMBEDDINGS_API_KEY=your-openai-api-key
OPENAI_EMBEDDINGS_MODEL=text-embedding-3-small

Example 4: Ollama (Local)

# LLM Provider
LLM_PROVIDER=ollama

# Ollama LLM Configuration
OLLAMA_LLM_BASE_URL=http://localhost:11434
OLLAMA_LLM_MODEL=llama2
OLLAMA_LLM_TEMPERATURE=0.0

# Embeddings Provider
EMBEDDINGS_PROVIDER=ollama

# Ollama Embeddings Configuration
OLLAMA_EMBEDDINGS_BASE_URL=http://localhost:11434
OLLAMA_EMBEDDINGS_MODEL=nomic-embed-text

B. Edit config.ini file

Configure your RAG backend API and evaluation settings:

MLflow Settings:

[mlflow]
tracking_uri = http://127.0.0.1:5000
experiment_name = RAG Evaluation
run_name = RAG Evaluation Run

Backend API Endpoint:

[endpoints]
# Your complete RAG API endpoint
response_api_path = https://your-app.com/api/respond

# HTTP method (POST or GET)
method = POST

# Request headers (comma-separated key:value pairs)
headers = Content-Type:application/json

# Request body fields (use {query} and {chat_id} as placeholders)
body_fields = prompt:{query},chat_id:{chat_id}

# JSON response keys
answer_key = response
contexts_key = context

Authentication:

[auth]
# Options: cookie, bearer, header, or none
type = cookie
cookie_name = session
cookie_value = your-session-cookie-value

Dataset:

[dataset]
path = test_dataset.csv
category = simple  # Options: simple (RAGAS) or guardrail (security)

C. Edit test_dataset.csv file

Add your test queries in the format: query,ground_truth,chat_id

Example:

query,ground_truth,chat_id
What is RAG?,RAG stands for Retrieval-Augmented Generation,1
How does vector search work?,Vector search finds similar items using embeddings,2

For detailed configuration help, see the comments in each config file.

3. Validate & Run

# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run

Results will be available in the MLflow UI at the configured tracking URI.

CLI Commands

# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel

Evaluation Categories

Set category in config.ini to choose evaluation type:

Simple (RAGAS Quality Metrics)

category = simple

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

Guardrail (Security Metrics)

category = guardrail

• Toxicity Score - Detects toxic content in responses
• Bias Score - Detects biased content in responses

Performance Metrics

Logged for all evaluation categories:

• Avg Response Time - Average API response time (ms)
• P90 Latency - 90th percentile latency (ms)
• Queries Per Second - Throughput (QPS)

Supported Providers

LLM Providers

Provider	Models	Configuration
Azure OpenAI	GPT-4, GPT-3.5, etc.	Requires API key, endpoint, deployment name
OpenAI	GPT-4, GPT-3.5, etc.	Requires API key
Google Gemini	Gemini 2.0 Flash, Gemini 1.5 Pro, etc.	Requires Google API key
Ollama	Llama 2, Mistral, etc.	Local installation, no API key needed

Embeddings Providers

Provider	Models	Configuration
Azure OpenAI	text-embedding-ada-002, etc.	Requires API key, endpoint, deployment name
OpenAI	text-embedding-3-small, text-embedding-3-large	Requires API key
HuggingFace	MiniLM-L6-v2, BGE, E5, etc.	Optional API key (local models supported)
Ollama	nomic-embed-text, etc.	Local installation, no API key needed

Changelog

Version 0.1.11 (2026-03-25)

New Features:

• ✨ Added support for Google Gemini LLM provider
• ✨ Added support for HuggingFace embeddings (including MiniLM-L6-v2)
• 📚 Enhanced documentation with provider examples

Dependencies:

• Added langchain-google-genai>=1.0.0
• Added langchain-huggingface>=0.0.1

Version 0.1.10

• Initial stable release with Azure, OpenAI, and Ollama support

License

MIT