RAGmortem

A debugger for RAG apps: it finds why an answer went wrong — retrieval, ranking, generation, or lack of evidence — and tests the diagnosis before you ship it.

Project Status: Open-source RAG debugger (v0.2.0)
RAGmortem is a fast, deterministic root-cause failure diagnoser for RAG pipelines. It isolates failures across retrieval, ranking, generation, and abstention without requiring an external LLM judge.

from ragmortem import diagnose, Chunk

# 20-second quickstart: diagnose an execution trace directly
result = diagnose(
    query="What is the mandatory engineering response time for a P1 incident?",
    retrieved_chunks=[{"id": "doc_dr", "text": "Disaster recovery drills occur quarterly.", "score": 0.14}],
    answer="Engineers must run quarterly disaster recovery exercises.",
    corpus=[
        Chunk(id="chunk_incident_severity_p1", text="P1 (Critical) incidents require acknowledgment within 15 minutes.", source="policy"),
        Chunk(id="chunk_dr_drill", text="Disaster recovery drills occur quarterly.", source="policy"),
    ],
    mode="corpus_aware",
)

print(result.to_text())

Output:

Failure: RETRIEVAL_SUSPECTED (suspected)
Evidence Score: 0.94

Why:
The answer becomes supported when the missing high-relevance chunk is available, but the relevant chunk was absent from the retrieved top-k results.

Evidence:
  * Retriever returned low-relevance chunks at runtime (top score: 0.140).
  * Corpus audit shows indexed documents exist with high relevance to question.
  * Corpus contains candidate 'chunk_incident_severity_p1' with distinct semantic relevance (score: 0.741).

Recommended next step:
Increase retrieval recall: inspect query reformulation, adjust embedding/indexing strategy, or expand candidate retrieval pool.

Mode: corpus_aware

---

1. What Problem RAGmortem Solves

When a RAG system returns an incorrect answer, traditional observability tools only compute continuous quality scores (e.g. faithfulness: 0.42). They do not tell you which component broke:

• Did the retriever fail to find the document? (retrieval_miss)
• Did candidate retrieval find it, but ranking dropped it below top-k? (ranking_miss)
• Did the generator receive the correct context, but ignore it or hallucinate? (generation_ignored_context)
• Was the question unanswerable from the knowledge base, but the model answered anyway? (should_abstain)
• Is telemetry insufficient to distinguish the cause? (unknown)

RAGmortem replaces ambiguous scores with deterministic, reproducible root-cause attribution and concrete next-step remediation actions.

---

2. Privacy & Data Safety

RAGmortem is built as a local-first developer debugging tool:

• Zero Telemetry: No tracking, no external analytics, no telemetry platform integration.
• No Network Calls: Diagnostic evaluation executes 100% locally on your CPU.
• No Stored API Keys: Diagnoser does not require, store, or transmit LLM API keys.
• No Prompts/Answers Uploaded: Your data stays on your machine.
• No Automatic Sensitive Logging: Document texts and queries are evaluated in-memory. Local trace persistence is strictly opt-in.

---

3. How It Works

RAGmortem uses an uncertainty-aware decision tree operating on observable execution telemetry:

1. Telemetry Completeness Check: Validates that query, answer, context, and similarity scores are present. Missing telemetry directly produces UNKNOWN rather than fabricated certainty.
2. Abstention & Refusal Probe: Detects whether the generator emitted a refusal. If low retrieval score matches refusal, marks NO_FAILURE (appropriate abstention). If context had strong scores, marks false refusal.
3. Candidate Cutoff Probe: Inspects the candidate retrieval pool prior to top-k truncation. If a viable candidate exists immediately below the cutoff ($\text{score gap} \le 0.08$), isolates a RANKING_SUSPECTED failure.
4. Context Score Probe: High similarity in context with an ungrounded or contradictory answer isolates GENERATION_SUSPECTED.
5. Corpus Audit vs. Trace-Only Boundary:
   • In corpus_aware mode, audits the corpus index to distinguish whether documents were missed (RETRIEVAL_SUSPECTED) or the question was unanswerable (ABSTENTION_SUSPECTED).
   • In trace_only mode, safely returns UNKNOWN because an APM trace without corpus access cannot infer document existence.

---

4. Installation & Quickstart

Requires Python >= 3.11.

Zero-Friction Instant CLI (No Installation Needed)

Run the bundled failure demonstration immediately using uvx:

uvx ragmortem demo

Inspect input trace schema and full JSON format:

uvx ragmortem schema

Install from PyPI

pip install ragmortem

For fast local NLI / cross-encoder semantic answer verification:

pip install "ragmortem[local-nli]"

Or install from GitHub:

pip install git+https://github.com/deveshkolte/RAGmortem.git

Or for local development:

git clone https://github.com/deveshkolte/RAGmortem.git
cd RAGmortem
pip install -e ".[dev,local-nli]"

CLI Quickstart

Check version:

ragmortem --version
# or: python -m ragmortem --version

Run the bundled demonstration on 5 synthetic failure cases with the screenshot-worthy Rich terminal UI:

ragmortem demo
# or instant zero-install:
uvx ragmortem demo

# Switch embedding calibration profile (minilm, openai, bge, relative)
ragmortem demo --profile relative

View input trace schema with interactive Rich syntax highlighting:

ragmortem schema

Diagnose your own execution traces:

# Trace-only diagnosis with screenshot-worthy Rich terminal UI
ragmortem diagnose trace.json

# Calibrate to your embedding model distribution
ragmortem diagnose trace.json --profile openai    # text-embedding-3-small/large
ragmortem diagnose trace.json --profile bge       # bge-large-en-v1.5
ragmortem diagnose trace.json --profile relative  # auto-calibrate to runtime score distribution

# Corpus-aware diagnosis
ragmortem diagnose traces.jsonl --corpus path/to/documents

# Export reports: HTML (dark-mode dashboard), structured JSON, or clean Markdown
ragmortem diagnose trace.json --format html --out report.html
ragmortem diagnose trace.json --format json --out report.json
ragmortem diagnose trace.json --format md --out report.md

---

5. Python SDK Usage

Trace Construction

Use ragmortem.trace to record runtime execution telemetry:

from ragmortem import trace, diagnose

t = trace(
    query="What is the P1 incident response SLA?",
    retrieved_chunks=[
        {"id": "doc_dr", "text": "Disaster recovery drills occur quarterly.", "score": 0.14},
        {"id": "doc_sla", "text": "Communication cadence is bi-weekly.", "score": 0.12},
    ],
    scores=[0.14, 0.12],
    answer="Engineers must run quarterly disaster recovery exercises.",
    context="[1] Disaster recovery drills occur quarterly.\n[2] Communication cadence is bi-weekly.",
)

Missing fields are never silently fabricated (missing scores, context, or answer remain explicit).

Diagnostic Operating Modes

RAGmortem supports two explicit diagnostic scopes (never silently switches between them):

Scope	Mode Parameter	Corpus Required?	Reference Answer?	What It Verifies
Trace-Only	trace_only	No	No	Only observable telemetry (scores, refusal, candidate pool). Returns UNKNOWN when evidence is ambiguous.
Corpus-Aware Observed	corpus_aware (no reference_answer)	Yes	No	Heuristic diagnosis: retrieval/ranking/generation/abstention suspicions based on corpus audit. Does not verify answer correctness.
Corpus-Aware Reference-Assisted	corpus_aware + reference_answer	Yes	Yes	Definitive diagnosis: compares generated answer to reference, searches retrieved/candidate chunks for supporting evidence. Returns NO_FAILURE, GENERATION_PROVEN, RETRIEVAL_MISS, RANKING_MISS, SHOULD_ABSTAIN, or UNKNOWN.

Trace-Only (Offline APM / Logs)

Zero knowledge-base dependencies. Evaluates observable trace telemetry directly:

result = diagnose(t, mode="trace_only")
print(result.failure_type)        # FailureType.UNKNOWN (safely avoids guessing)
print(result.recommended_action) # Explains missing corpus access

Corpus-Aware (Knowledge Base Probing)

Probes the corpus index to disambiguate retrieval vs. answerability failures. Accepts three input formats:

1. Directory path of markdown files:

result = diagnose(t, corpus="path/to/markdown/documents", mode="corpus_aware")

2. In-memory list of Chunk objects (or dicts):

from ragmortem import Chunk

corpus = [
    Chunk(id="doc_p1", text="P1 incidents require 15-minute response.", source="policy"),
    Chunk(id="doc_dr", text="Disaster recovery drills occur quarterly.", source="policy"),
]
result = diagnose(t, corpus=corpus, mode="corpus_aware")

3. Custom retriever adapter (.retrieve(query, k)):

# Any object implementing .retrieve(query: str, k: int) -> tuple[list[Chunk], list[float]]
result = diagnose(t, corpus=my_custom_retriever, mode="corpus_aware")

With reference answer (definitive diagnosis):

result = diagnose(
    t,
    corpus=corpus,
    mode="corpus_aware",
    reference_answer="P1 incidents require 15-minute response.",
)

Key distinction: Without reference_answer, corpus_aware mode uses heuristics (corpus audit + score thresholds) and returns suspected failure types (GENERATION_SUSPECTED, RETRIEVAL_SUSPECTED, etc.). It cannot verify whether the generated answer is factually correct — it only flags when high-relevance context was present but the answer may be wrong. With reference_answer, the diagnosis becomes definitive (NO_FAILURE, GENERATION_PROVEN, RETRIEVAL_MISS, etc.) by comparing the answer to the reference and searching the corpus for supporting evidence.

Complete Integration Demonstrations

• examples/integrations/minimal_rag.py: Minimal code integration across all 6 diagnostic scenarios.
• examples/integrations/realistic_rag_demo.py: End-to-end small RAG app smoke test (documents → embeddings → retrieval → context → generator → RAGmortem).

Run the demo locally:

python examples/integrations/realistic_rag_demo.py

---

6. Hijack Existing Workflows (The Growth Wedge)

1-Line LangChain & LlamaIndex Callbacks

Drop RAGmortem directly into your existing chain or query engine with one line of code:

LangChain Callback

from ragmortem import RAGmortemCallbackHandler

handler = RAGmortemCallbackHandler(auto_diagnose=True)
chain.invoke(
    {"query": "What is the P1 incident SLA?"},
    config={"callbacks": [handler]},
)
# Or manually inspect diagnosis:
result = handler.diagnose()

LlamaIndex Callback

from ragmortem import LlamaIndexRAGmortemHandler
from llama_index.core.callbacks import CallbackManager
from llama_index.core import Settings

handler = LlamaIndexRAGmortemHandler(auto_diagnose=True)
Settings.callback_manager = CallbackManager([handler])

response = query_engine.query("What is the P1 incident SLA?")
result = handler.diagnose()

---

Zero-Dependency Framework Adapters (from_langchain & from_llamaindex)

If you don't use callbacks or prefer to convert existing run objects, convert framework outputs or run dictionaries directly into standardized Trace instances with zero hard dependencies (neither langchain nor llama-index needs to be installed):

LangChain Adapter (from_langchain)

from ragmortem import from_langchain, diagnose

# Duck-types RetrievalQA output dicts, chain.invoke() runs, or Document objects
run_output = chain.invoke({"query": "What is the P1 incident SLA?"})
trace_obj = from_langchain(run_output)

result = diagnose(trace_obj)
print(result.to_text())

LlamaIndex Adapter (from_llamaindex)

from ragmortem import from_llamaindex, diagnose

# Duck-types Response / NodeWithScore objects or query engine output dicts
response = query_engine.query("What is the P1 incident SLA?")
trace_obj = from_llamaindex(response)

result = diagnose(trace_obj)
print(result.to_text())

---

OpenTelemetry, Phoenix & LangSmith Trace Importers

Import traces directly from Arize Phoenix, LangSmith, or generic OpenTelemetry spans:

# Auto-detects schema and diagnoses directly
ragmortem diagnose phoenix_spans.json
ragmortem diagnose langsmith_run.json

# Or import into standardized RAGmortem trace format:
ragmortem import phoenix_spans.json --out trace.json --diagnose

In Python:

from ragmortem import import_trace, diagnose

trace_obj = import_trace("path/to/otel_spans.json")
result = diagnose(trace_obj)
print(result.to_text())

---

pytest-ragmortem: Auto-Diagnosis in CI & ragmortem_assert

RAGmortem includes a built-in pytest plugin (pytest-ragmortem) that intercepts test failures and isolates component root causes automatically:

1. Explicit Assertion Helper (ragmortem_assert)

Assert that a RAG execution succeeded without failure. If any failure is diagnosed (RETRIEVAL_SUSPECTED, RANKING_SUSPECTED, GENERATION_SUSPECTED, etc.), it formats and prints a Rich diagnostic card and raises an AssertionError with remediation recommendations:

from ragmortem import ragmortem_assert

def test_production_qa():
    run_output = my_rag_app.query("What is the mandatory P1 response SLA?")
    
    # Diagnoses trace; if failure detected, raises AssertionError with Rich diagnostic card
    ragmortem_assert(
        trace_input=run_output,
        reference_answer="15 minutes",
        profile="minilm",  # or "openai", "bge", "relative"
    )

2. Automatic Diagnosis via rag_trace Fixture

When an ordinary unit test assertion fails in CI, rag_trace automatically intercepts the test failure, isolates the root cause from execution telemetry, and outputs a formatted failure card:

# test_rag_pipeline.py
def test_p1_sla(rag_trace):
    res = rag_app.query("What is the P1 response time?")
    rag_trace.record(
        query="What is the P1 response time?",
        retrieved_chunks=res.retrieved_chunks,
        scores=res.scores,
        answer=res.answer,
    )
    # If this assertion fails, pytest-ragmortem automatically prints the root-cause diagnosis!
    assert "15 minutes" in res.answer

In GitHub Actions, pytest-ragmortem automatically:

1. Emits inline workflow annotations (::error file=...,title=RAGmortem Diagnosis [GENERATION_SUSPECTED]...)
2. Generates an executive diagnostic table in $GITHUB_STEP_SUMMARY.

---

7. CLI: Diagnosing Saved Traces

Diagnose saved JSON trace files from the terminal:

# Trace-only mode (offline APM logs, no corpus needed)
ragmortem diagnose-trace trace.json

# Corpus-aware mode (with a documents directory)
ragmortem diagnose-trace trace.json --corpus path/to/documents

# Structured JSON output
ragmortem diagnose-trace trace.json --corpus path/to/documents --json

Standard JSON Trace Schema

{
  "query": "What is the mandatory engineering response time for a P1 incident?",
  "retrieved_chunks": [
    {
      "id": "chunk_dr_drill_frequency",
      "text": "Disaster recovery drills occur quarterly.",
      "score": 0.1407
    }
  ],
  "scores": [0.1407],
  "answer": "Engineers must execute drills quarterly.",
  "context": "[1] Disaster recovery drills occur quarterly.",
  "metadata": {
    "model": "qwen-2.5",
    "latency_ms": 320.5
  }
}

---

---

8. Failure Taxonomy

Failure Mode	Canonical Type	What Happened	Recommended Next Step
retrieval_suspected	retrieval_miss	Supporting chunk exists in corpus but was omitted from candidate set.	Increase retrieval recall, adjust embedding model, or expand candidate pool.
ranking_suspected	ranking_miss	Supporting chunk was retrieved in candidates, but fell below top-k cutoff.	Tune re-ranker, expand context window (top-k), or re-score candidates.
generation_suspected	generation_ignored_context	High-relevance context was in prompt, but generator hallucinated or ignored it. Observed mode only — suspected, not proven.	Tighten prompt grounding constraints, lower temperature, refine prompt.
abstention_suspected	should_abstain	Corpus contains no supporting evidence, but generator fabricated an answer.	Add strict refusal instructions for out-of-domain queries; filter low scores.
unknown	unknown	Telemetry is incomplete (missing scores, dropped candidates) or signals are ambiguous.	Provide complete telemetry or supply corpus index access.
no_failure	no_failure	Reference-assisted mode only: Answer is verified grounded, or system correctly refused unanswerable question.	No corrective action required.
generation_proven	generation_ignored_context	Reference-assisted mode only: Reference answer found in retrieved context, but generator produced a different answer.	Tighten prompt grounding constraints, lower temperature, refine prompt.
retrieval_miss	retrieval_miss	Reference-assisted mode only: Reference answer exists in corpus but was not retrieved.	Increase retrieval recall, adjust embedding model, or expand candidate pool.
ranking_miss	ranking_miss	Reference-assisted mode only: Reference answer found in candidate pool but ranked below top-k.	Tune re-ranker, expand context window (top-k), or re-score candidates.
should_abstain	should_abstain	Reference-assisted mode only: Reference answer not in corpus; system should have abstained.	Add strict refusal instructions; filter low scores.

Note: generation_suspected, retrieval_suspected, abstention_suspected, ranking_suspected are heuristic suspicions returned by observed modes (trace-only and corpus-aware without reference_answer). The definitive types generation_proven, retrieval_miss, ranking_miss, should_abstain, no_failure are only returned by reference-assisted mode (corpus_aware + reference_answer).

---

9. Validation Benchmarks & Historical Evidence

To ensure scientific credibility, RAGmortem distinguishes strictly between controlled benchmarks, adversarial holdouts, and integration smoke tests.

Important: Controlled benchmark metrics do NOT represent real-world accuracy on uncurated production data.

Evaluation Suite / Benchmark	Purpose	Corpus-Aware Accuracy	Trace-Only Accuracy	Unknown Recall	Key Findings
Day 5 Controlled In-Sample (122 cases)	Development baseline	100.0% (122/122)	N/A	N/A	Verified taxonomy and heuristic logic coherence under ideal conditions.
Day 6 Adversarial Holdout (120 cases)	Cloud/DevOps holdout	90.8% (109/120)	58.3% (70/120)	100.0% (20/20)	Exposed fixed-threshold brittleness; motivated uncertainty-aware scoring.
Day 7/8 BioTrial Frozen Holdout (100 cases)	Clinical trials holdout	100.0% (frozen upper-bound) / 96.0% overall	60.0% (50.0% resolved, 100% precision)	100.0% (20/20)	100% recall on ranking misses and generation failures. Trace-only safely refuses low-score ambiguity.
Day 8 FinDebt Domain Holdout (40 cases)	Complex corporate finance	65.0% overall / 83.3% resolved	7.5% (safely refuses)	100.0%	Demonstrated that background terminology traps in specialized domains require lexical support analysis.
Integration Smoke Test (realistic_rag_demo.py)	End-to-end integration	Smoke validation	Smoke validation	Smoke validation	Proves public API usability in real Python code without synthetic fault injection.

---

10. What Each Mode Can Truthfully Claim

Diagnostic Mode	What It Can Truthfully Claim	What It Cannot Claim
trace_only	Operates strictly on provided runtime telemetry (query, retrieved chunks, scores, candidates, answer). Flags candidate truncation (RANKING_SUSPECTED), high-score context ignored (GENERATION_SUSPECTED), and appropriate refusal (NO_FAILURE).	Cannot verify whether unretrieved documents exist in your corpus or if the answer is factually correct. When retrieval scores are low or borderline, it safely refuses to guess and outputs UNKNOWN.
corpus_aware	Audits your corpus index to determine if high-relevance documents exist for the query. Distinguishes retrieval misses (RETRIEVAL_SUSPECTED) from unanswerable queries (ABSTENTION_SUSPECTED), including terminology trap detection.	Cannot verify factual correctness of the generated answer without an expected reference answer. Diagnoses retrieval quality and grounding alignment, not semantic truth.
reference_assisted	Verifies lexical correctness against a user-supplied reference answer. Determines if evidence was present in context (GENERATION_PROVEN), candidate pool (RANKING_MISS), broader corpus (RETRIEVAL_MISS), or absent (SHOULD_ABSTAIN).	Reference matching is purely lexical, not semantic. Paraphrased or synonymous answers may not match. GENERATION_PROVEN means lexical containment evidence, not mathematical proof of reasoning.
oracle	Runs counterfactual replay isolating retriever vs. generator ceilings in controlled research benchmarks.	Cannot run on uncurated production traces without a known ground-truth gold chunk.

---

11. Known Limitations

• Embedding Distribution Calibration: RAGmortem includes distribution-aware auto-calibration (ScoreCalibrator) supporting OpenAI (text-embedding-3-small/large), Cohere (embed-v3), BGE (bge-large-en-v1.5), and MiniLM baselines. When evaluating unknown embedding distributions, empirical distribution calibration automatically analyzes observed candidate pools (mean, spread, z-scores) rather than relying exclusively on fixed thresholds.
• Semantic Verification vs. Token Matching: Answer verification defaults to fast deterministic token matching and seamlessly upgrades to local NLI / Cross-Encoder entailment verification when installed via pip install ragmortem[local-nli], enabling robust semantic verification for paraphrased responses.
• GENERATION_PROVEN Means Lexical Evidence, Not Proof: The enum label GENERATION_PROVEN denotes that the reference answer text was verbatim contained within the retrieved context chunks, yet the generator failed to produce it. It does not represent formal mathematical or logical proof.
• Counterfactual Replay Requires Gold Chunk: Counterfactual replay only runs in oracle mode, which strictly requires a verified gold_chunk_id.
• Synthetic Benchmark Labels: All labels in the project's evaluation datasets (evals/) were generated synthetically using the project's own fault injector. No real-world production outages or third-party datasets have been evaluated yet. See docs/EVIDENCE.md for full provenance and tuning history.
• Single-Hop Factual Focus: Optimized for single-chunk answerable lookups. Multi-hop reasoning across disjoint documents remains active research.
• Uncalibrated Confidence: evidence_score represents relative deterministic heuristic strength, not a calibrated Bayesian probability.

---

12. Testing

Run the full offline test suite:

pytest tests/ -v

All 192 tests run 100% offline on CPU in ~10 seconds without network requests or API keys.

---

13. License

Apache-2.0 License. See LICENSE for details.