rag-eval

Score any RAG pipeline on retrieval precision and answer faithfulness — zero framework lock-in, zero required API keys.

---

Demo

---

Install in 30 seconds

Install the core library and CLI:

pip install rag-eval-py

Optional extras:

# FastAPI server + uvicorn
pip install "rag-eval-py[api]"

# OpenAI LLM-as-a-judge backend
pip install "rag-eval-py[judge]"

# All extras
pip install "rag-eval-py[api,judge]"

---

Quickstart

Evaluate samples programmatically in 10 lines of Python:

from rag_eval.faithfulness import faithfulness
from rag_eval.retrieval import precision_at_k

# 1. Provide your RAG outputs
retrieved = ["Paris is the capital of France.", "Lyon is in France."]
golden = ["Paris is the capital of France."]
answer = "Paris is the capital of France."
context = ["Paris is the capital and largest city of France."]

# 2. Score retrieval and answer faithfulness
p_at_k = precision_at_k(retrieved, golden, k=1)
score, supported, unsupported = faithfulness(answer, context)

print(f"Retrieval Precision@1: {p_at_k:.2f}")  # 1.00
print(f"Answer Faithfulness:    {score:.2f}")  # 1.00

---

How Scoring Works

rag-eval evaluates two independent axes of RAG performance:

1. Retrieval Quality: Precision@k

Measures whether the retriever fetched ground-truth chunks in its top-$k$ results:

$$\text{Precision@k} = \frac{|\text{set}(\text{retrieved}[:k]) \cap \text{set}(\text{golden_documents})|}{k}$$

• Uses strict set semantics: duplicated golden documents never artificially inflate the score.
• If no retrieval data is supplied, this metric is omitted (None).

2. Answer Faithfulness: Claim Overlap

Evaluates whether the generator hallucinated information beyond the retrieved context:

1. Claim Extraction: The answer is decomposed into atomic claims by splitting on sentence punctuation ([.!?]) and contrasting clauses (, but , ; , etc.).
2. Stopword Filtering: Extracts alphanumeric content words while stripping English stopwords ("the", "is", "of", etc.).
3. Threshold Check: A claim is verified as supported if at least 80% (SUPPORT_THRESHOLD = 0.8) of its content words appear in the context.
4. Faithfulness Score: $$\text{Faithfulness} = \frac{|\text{supported claims}|}{|\text{total claims}|}$$

If the answer has no claims, the score defaults to 1.0.

---

JSONL Format Spec

Feed evaluation data in plain .jsonl files (one JSON object per line):

Field	Type	Required?	Meaning
question	string	Yes	The user prompt or question passed to the RAG pipeline.
answer	string	Yes	The final answer generated by the LLM.
context	list[string]	Yes (min 1)	The actual text chunks injected into the LLM prompt context.
golden_documents	list[string]	No	Ground-truth reference chunks (required for retrieval scoring).
retrieved	list[string]	No	Chunks returned by retriever in ranked order.
id	string	No	Unique sample ID (auto-generates UUID4 if omitted).

---

CLI Reference

rag-eval eval [OPTIONS] DATA

Option / Flag	Type	Default	Description
DATA	Path	Required	Path to the evaluation .jsonl file.
--json	Flag	False	Render output report as a JSON string to stdout.
--skip-errors	Flag	False	Skip invalid lines and log warnings instead of aborting.
--k	int	5	Rank cutoff for Precision@k retrieval calculation.
--claims-only	Flag	False	Skip retrieval metrics and evaluate answer faithfulness only.
--judge	string	None	Optional LLM judge backend (openai).
--help	Flag		Show help message and exit.

---

FastAPI Usage

Run the bundled evaluation microservice:

uvicorn examples.fastapi_server:app --reload --port 8000

Endpoints

• GET /health
  Returns {"status": "ok"}

• POST /evaluate
  Accepts a JSON payload with samples and optional k:

  {
    "samples": [
      {
        "question": "What is the capital of France?",
        "answer": "Paris is the capital of France.",
        "context": ["Paris is the capital and largest city of France."],
        "golden_documents": ["Paris is the capital and largest city of France."],
        "retrieved": ["Paris is the capital and largest city of France."],
        "id": "sample-1"
      }
    ],
    "k": 5
  }
  

  Returns EvalReport JSON with sample statistics and arithmetic means.

---

Framework Adapters

rag-eval is 100% framework-agnostic. Pre-built adapters in examples/ map outputs from popular frameworks without introducing hard dependencies:

• Haystack 2.x: examples/haystack_adapter.py — maps pipeline outputs directly into list[Sample].
• LangChain: examples/langchain_adapter.py — maps LCEL and RetrievalQA datasets into list[Sample].

---

Limitations

• Heuristic Claim Overlap: The default zero-dependency claim_overlap method uses lexical content-word overlap ($0.8$ threshold). While extremely fast and completely free of API charges, it may miss semantic paraphrasing or complex negations.
• When to Use --judge openai: For mission-critical production evaluations requiring semantic nuance, pass --judge openai to verify claims using GPT-4o-mini or specify your preferred LLM judge.

---

Contributing & License

Contributions are welcome! To run tests, linting, and type checking locally:

hatch run test
hatch run lint
hatch run typecheck

Released under the MIT License. Copyright (c) 2026 Mustafa Abad Shah.