rag-sentinel 0.1.8

RAG Evaluation Framework using Ragas metrics and MLflow tracking

RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

1. Create Virtual Environment

# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Package

pip install rag-sentinel

Quick Start

1. Initialize Project

rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - App settings and authentication
• rag_eval_config.yaml - Master configuration
• test_dataset.csv - Sample test dataset

2. Configure

Edit .env file:

• Add your LLM API keys (Azure OpenAI, OpenAI, or Ollama credentials)
• Set API endpoints and deployment names

Edit config.ini file:

• Set your RAG backend URL in app_url
• Set API paths for context and answer endpoints (context_api_path, answer_api_path)
• Configure authentication (cookie, bearer token, or API key)
• Set MLflow tracking URI (default: http://127.0.0.1:5000)

Edit test_dataset.csv file:

• Add your test queries in format: query,ground_truth,chat_id
• Example: What is RAG?,RAG stands for Retrieval-Augmented Generation,1

For detailed configuration help, see the comments in each config file.

3. Validate & Run

# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run

Results will be available in the MLflow UI at the configured tracking URI.

CLI Commands

# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel

Evaluation Categories

Set category in config.ini to choose evaluation type:

Simple (RAGAS Quality Metrics)

category = simple

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

Guardrail (Security Metrics)

category = guardrail

• Toxicity Score - Detects toxic content in responses
• Bias Score - Detects biased content in responses

Performance Metrics

Logged for all evaluation categories:

• Avg Response Time - Average API response time (ms)
• P90 Latency - 90th percentile latency (ms)
• Queries Per Second - Throughput (QPS)

License

MIT