RAG Debugger 🔍

Visualize exactly where your RAG pipeline breaks.

Most RAG problems are invisible — wrong chunks retrieved, LLM given bad context, silent failures at retrieval. RAG Debugger makes every step visible so you can fix it fast.

QUERY  "What is the cancellation policy?"

  ✏️  Query Rewriter    success    0.014ms
  🔍  Vector Search     success    0.025ms    3 chunks  top score 0.94
  🤖  GPT-4o            error      0.012ms    ← broke here

💥 Pipeline broke at: llm
   RuntimeError: OpenAI API rate limit exceeded

What It Does

• Traces every step — query transform, embedding, retrieval, rerank, LLM
• Shows your chunks — see exactly what context your LLM received
• Catches breaks — know immediately which stage failed and why
• Visual UI — pipeline flowchart at localhost:7384, opens automatically
• Terminal output — rich summary printed after every trace
• Works with anything — LangChain, LlamaIndex, or raw Python

Install

pip install rag-debugger

With framework support:

pip install rag-debugger[langchain]
pip install rag-debugger[llama-index]
pip install rag-debugger[all]

Quickstart

Generic pipeline (works with anything)

from rag_debugger import init, StepType, Chunk

debugger = init()   # starts UI at localhost:7384

with debugger.trace("What is the refund policy?") as t:

    with debugger.step(StepType.RETRIEVAL, "My Retriever") as s:
        chunks = my_retriever.search(t.query)
        s.chunks = chunks

    with debugger.step(StepType.LLM, "GPT-4o") as s:
        answer = my_llm.generate(t.query, chunks)
        s.model = "gpt-4o"

    t.answer = answer

LangChain

from rag_debugger import init

debugger = init(framework="langchain")

# pass handler to any LangChain component
chain.invoke(
    {"query": "What is the refund policy?"},
    config={"callbacks": [debugger.handler]}
)

Decorator style

from rag_debugger import init, StepType

debugger = init()

@debugger.trace_step(StepType.RETRIEVAL, "My Retriever")
def retrieve(query):
    return vector_store.search(query)

What You See

Terminal

╭─────────────────────────────────────────╮
│  RAG Debugger — Trace a1b2c3d4          │
╰─────────────────────────────────────────╯

QUERY  "What is the refund policy?"

  ✏️   Query Rewriter    success    0.014ms
  🔍   Vector Search     success    0.025ms    3 chunks  top score 0.94
  🤖   GPT-4o            success    0.012ms    312→52 tokens

ANSWER  "You have 30 days to request a refund."

Total: 0.051ms  |  framework: custom  |  steps: 3

Browser UI — localhost:7384

• Left panel — all traces with status and timing
• Right panel — full pipeline flowchart
• Click any step — see chunks, tokens, errors, input/output
• Live updates — new traces appear instantly as pipeline runs

How It Works

Your RAG pipeline
      │
      ▼
import rag_debugger       ← wraps each stage
      │
      ├── records every step (timing, chunks, tokens, errors)
      │
      ├── terminal summary (rich)
      │
      └── FastAPI server  ← localhost:7384
              │
              └── pipeline flowchart UI

No external services. No API keys. Runs entirely on your machine.

Examples

# see a working pipeline
python examples/basic_usage.py

# see the debugger catching errors
python examples/broken_pipeline.py

Project Structure

rag_debugger/
├── core/           ← tracer, store, models
├── integrations/   ← langchain, llama_index, generic
├── server/         ← fastapi + websocket
├── ui/             ← browser interface
└── cli/            ← terminal output

Contributing

Pull requests welcome. See CONTRIBUTING.md to get started.

Areas that need work:

• Haystack integration
• LlamaIndex deeper event coverage
• Embedding step visualization
• Trace export to JSON/CSV

License

MIT