rag-sentinel 0.1.1

RAG Evaluation Framework using Ragas metrics and MLflow tracking

RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

Installation

pip install rag-sentinel

Quick Start

1. Initialize Project

mkdir my-rag-eval
cd my-rag-eval
rag-sentinel init

This creates:

• .env - LLM/Embeddings API keys
• config.ini - App settings and authentication
• rag_eval_config.yaml - Master configuration

2. Configure

Edit .env:

LLM_PROVIDER=azure
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com/

Edit config.ini:

[app]
app_url = https://your-rag-app.com/backend

[auth]
type = cookie
cookie_name = session
cookie_value = your-session-cookie

3. Create Test Dataset

Create test_dataset.csv:

query,ground_truth,chat_id
Hello,Hello! How can I help you?,1

4. Run Evaluation

rag-sentinel run

View results at: http://127.0.0.1:5001

CLI Commands

rag-sentinel init              # Initialize project
rag-sentinel run               # Run evaluation (auto-starts MLflow)
rag-sentinel run --no-server   # Run without starting MLflow
rag-sentinel validate          # Validate configuration

Metrics

• Faithfulness - Factual consistency of answer with context
• Answer Relevancy - How relevant the answer is to the question
• Context Precision - Quality of retrieved context
• Answer Correctness - Comparison against ground truth

License

MIT