The Problem

RAG pipelines fail silently. You get a wrong answer and have no idea if it's a chunking problem, a retrieval miss, a position bias, or a hallucination. Existing evaluation tools give you a score — they don't tell you why.

rag-doctor tells you why.

══════════════════════════════════════════════════════════════
  RAG-DOCTOR ✗ ISSUES FOUND
══════════════════════════════════════════════════════════════
  Root Cause : context_position_bias (RC-2)
  Severity   : HIGH
  Finding    : Best document at position 1/2 — in danger zone (risk: 1.00)
──────────────────────────────────────────────────────────────
  Fix: Enable a reranker to push the most relevant document to position 0.
  Config Patch: {"retrieval.reranker": true}
══════════════════════════════════════════════════════════════

Quick Start

pip install rag-doctor

from rag_doctor import Doctor
from rag_doctor.connectors.mock import MockConnector

connector = MockConnector(corpus=[
    {"id": "doc1", "content": "For liver disease patients maximum acetaminophen dose is 2000mg per day."},
    {"id": "doc2", "content": "Standard adult dose: up to 4000mg per day."},
])

docs   = connector.retrieve("acetaminophen dose liver disease", top_k=3)
answer = "The maximum daily dose is 4000mg."

report = Doctor.default().diagnose(
    query    = "What is the max acetaminophen dose for liver disease?",
    answer   = answer,
    docs     = docs,
    expected = "For liver disease patients max dose is 2000mg per day.",
)
print(report.to_text())

How It Works

rag-doctor runs a deterministic six-tool agent loop. Each tool targets a specific failure mode:

No Database. No LLM. No API Keys.

rag-doctor builds an ephemeral VectorStore in memory from whatever documents you pass. Your production database is never touched.

Embedding backends (auto-selected, no config needed):

Three Ways to Use It

Mode A — Debug from Logs (no re-query needed)

from rag_doctor import Doctor
from rag_doctor.connectors.base import Document

docs = [
    Document(content=row["text"], score=row["score"], position=i)
    for i, row in enumerate(db_rows)
]
report = Doctor.default().diagnose(
    query="What is the refund policy?",
    answer="30 days for all customers.",
    docs=docs,
    expected="Enterprise customers get 90-day refunds.",
)
print(report.root_cause)      # retrieval_miss
print(report.fix_suggestion)  # Increase top_k or check corpus coverage.

Mode B — Corpus-Level Evaluation (CI / pytest)

connector = MockConnector(corpus=YOUR_CORPUS)
docs      = connector.retrieve(query, top_k=5)
report    = Doctor.default(connector).diagnose(query=query, answer=answer, docs=docs, expected=expected)
assert report.severity in ("low", "medium"), report.to_text()

Mode C — Connect Your Production Stack

class ChromaConnector(PipelineConnector):
    def retrieve(self, query, top_k=5):
        results = self.collection.query(query_texts=[query], n_results=top_k)
        return [Document(content=d, score=s, position=i) for i,(d,s) in enumerate(...)]

report = Doctor.default(ChromaConnector(my_collection)).diagnose(...)

CLI Reference

# Single query
rag-doctor diagnose \
  --query    "What is the termination notice?" \
  --answer   "30 days." \
  --expected "Enterprise requires 90 days written notice."

# Batch from JSONL
rag-doctor batch --input examples/batch_example.jsonl --fail-on-severity high

# JSON output for CI
rag-doctor diagnose --query "..." --answer "..." --output json | jq .root_cause

Local Setup (Mac)

git clone https://github.com/your-org/rag-doctor
cd rag-doctor
chmod +x scripts/test_local_mac.sh
./scripts/test_local_mac.sh

Documentation

License

MIT — free for personal and commercial use.