rag-lens

An open-source, pip-installable Python library for quantitatively evaluating RAG pipelines using LLM-as-a-Judge.

"I don't just build RAG systems — I built the tool that measures whether RAG systems work."

Why This Exists

RAG systems are easy to prototype and hard to trust. Most teams evaluate them by manually reading a handful of outputs and eyeballing whether the answers "look right." This breaks down at scale, doesn't catch regressions, and can't tell you why a pipeline underperforms.

RAG-Eval Framework replaces vibes-based evaluation with a scored, reproducible pipeline that separates retrieval quality from generation quality.

Quickstart

pip install ram-rag-lens
cp .env.example .env  # Add your GROQ_API_KEY

from rag_lens.evaluators import Faithfulness, AnswerRelevance
from rag_lens.judge import GroqJudge
from rag_lens.ingestion import QAItem, RAGOutput

judge = GroqJudge()

output = RAGOutput(
    item_id="test_001",
    question="What is RAG?",
    retrieved_context=["RAG combines retrieval with generation to reduce hallucinations."],
    generated_answer="RAG is a technique that retrieves documents before generating an answer.",
)

score = Faithfulness(judge=judge).score(output)
print(score.score)      # → 0.95
print(score.rationale)  # → "The answer is fully grounded in the retrieved context."

The 4 Metrics

All scores in [0.0, 1.0]. All scored by an LLM judge (Llama-3 via Groq).

CLI Usage

# Evaluate a single RAG output log
rag-lens evaluate --dataset data/qa_dataset.json --outputs my_rag_outputs.jsonl

# Run the full 18-configuration benchmark sweep
rag-lens benchmark --matrix data/configs/benchmark_matrix.yaml

# Regenerate leaderboard + charts from existing results (no API calls)
rag-lens report --results results/raw/

18-Configuration Benchmark

The framework ships a benchmark that tests 18 RAG configurations against a 100-question dataset:

Run it:

python benchmarks/run_18_config_benchmark.py

Judge Validation

LLM-as-a-judge scores are estimates, not ground truth. Before trusting benchmark conclusions, we validate the judge against human labels on a 20–30 item subset. See docs/qa-testing.md §4 for the methodology. Agreement rates are documented honestly — not hidden.

⚠️ Data notice: This tool sends dataset and RAG output text to the Groq API to power the judge. Do not use with sensitive or confidential data.

Project Structure

See docs/project-structure.md for a full annotated layout.

src/rag_lens/
├── judge/        → BaseJudge + GroqJudge (swappable LLM backend)
├── evaluators/   → Faithfulness, AnswerRelevance, ContextPrecision, ContextRecall
├── ingestion/    → Pydantic schemas + dataset loader
├── retrieval/    → 3 chunkers + 3 embedding wrappers + 2 retrievers
├── pipeline/     → RunConfig + benchmark runner (with resume support)
└── reporting/    → Leaderboard CSV + 4 matplotlib charts

Installation (from source)

git clone https://github.com/shriram1206/rag-lens-framework.git
cd rag-lens-framework
pip install -e ".[dev]"
cp .env.example .env  # Fill in GROQ_API_KEY
pytest tests/unit tests/integration  # All tests use mocked judge — free to run

Tech Stack

Python 3.11+ · Groq (Llama-3.1-70B) · ChromaDB · sentence-transformers · rank_bm25 · Pydantic v2 · pandas · matplotlib · typer · tenacity

Author

Shriram M · GitHub · LinkedIn · shriram.coder@gmail.com

MIT License