RAG Debugger

Your RAG chatbot is wrong.

But why?

RAG Debugger finds:

• bad retrieval
• irrelevant context
• hallucinated answers
• missing information

Install

pip install rag-debug-cli

Or from source, for development:

git clone https://github.com/Perevoznyi-Creation/rag_debugger.git
cd rag_debugger
pip install -e ".[dev]"

Usage

from rag_debugger import debug_rag

report = debug_rag(
    question="What is the refund policy?",
    retrieved_documents=[
        "Customers can return products within 30 days.",
        "Warranty lasts for 2 years.",
    ],
    answer="Customers can return products within 30 days.",
)

print(report)

RAG Score: 85/100

Retrieval:
  ✓ (0.78) Customers can return products within 30 days.
  ✗ (0.31) Warranty lasts for 2 years.

Grounding:
  ✓ Answer supported by retrieved context

Problems:
  ⚠ Irrelevant document retrieved: Warranty lasts for 2 years.

Recommendation:
  Reduce retrieval k from 2 to 1

CLI

rag-debug analyze examples/basic.json
rag-debug analyze examples/basic.json --html report.html
rag-debug evaluate examples/customer_questions.json
rag-debug evaluate examples/customer_questions.json --save before.json
rag-debug evaluate examples/customer_questions_v2.json --save after.json
rag-debug compare before.json after.json

Input JSON shape (single case):

{
  "question": "How long are refunds?",
  "documents": ["Refunds take 5 days"],
  "answer": "Refunds take 30 days"
}

Dataset JSON shape (for evaluate, a list of cases, label optional):

[
  {
    "label": "refund timing",
    "question": "How long are refunds?",
    "documents": ["Refunds take 5 days"],
    "answer": "Refunds take 30 days"
  }
]

compare diffs two evaluate --save result files by matching case labels — useful for checking whether a RAG change (chunk size, top_k, prompt) actually helped:

Before: 38%
After:  53%
Change: +15%

Improvements:
  warranty question (bad retrieval): 13 -> 72 (+59)

Choosing an LLM judge

Grounding/hallucination checks are done by an LLM judge, pluggable per provider. Set one API key and, optionally, pick the provider:

export GROQ_API_KEY="..."        # free tier — default if RAG_DEBUGGER_JUDGE is unset
export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."

export RAG_DEBUGGER_JUDGE="groq"                # uses that provider's default model
export RAG_DEBUGGER_JUDGE="groq:<model-name>"   # or pick a specific model
export RAG_DEBUGGER_JUDGE="openai:<model-name>"
export RAG_DEBUGGER_JUDGE="anthropic:<model-name>"

Each provider's default model is defined in src/rag_debugger/judges/<provider>_judge.py (DEFAULT_MODEL) — check there or your provider's docs for current model names, since they change over time and this README intentionally doesn't pin one.

Or override per call/command:

debug_rag(question=..., retrieved_documents=..., answer=..., judge="openai:<model-name>")

rag-debug analyze examples/basic.json --judge anthropic:<model-name>

OpenAI and Anthropic support are optional extras (pip install -e ".[openai]", ".[anthropic]", or ".[all-judges]" for both) — only the SDK for the provider you actually use needs to be installed.

For local development, put your key(s) in a .env.local file (gitignored) — both examples/basic.py and the CLI load it automatically.

LangChain integration

DebugRetriever wraps any existing BaseRetriever — it's a real Runnable, so it drops directly into an LCEL chain with no other changes:

from rag_debugger.integrations.langchain import DebugRetriever

retriever = DebugRetriever(retriever=my_vectorstore.as_retriever())

chain = retriever | prompt | llm
answer = chain.invoke("What is the refund policy?")

report = retriever.debug(answer)
print(report)

DebugRetriever remembers the query and documents from the most recent invoke(); .debug(answer) runs debug_rag() against them. Pass judge= the same way as debug_rag() (a spec string or Judge instance) if you don't want the RAG_DEBUGGER_JUDGE default. See examples/langchain_integration.py for a runnable end-to-end example.

Requires the langchain extra: pip install -e ".[langchain]".

LlamaIndex integration

Same pattern, for LlamaIndex's BaseRetriever:

from rag_debugger.integrations.llama_index import DebugRetriever

retriever = DebugRetriever(retriever=index.as_retriever())

nodes = retriever.retrieve("What is the refund policy?")
# ... build a response from `nodes` however your pipeline does ...
answer = "Customers can return products within 30 days."

report = retriever.debug(answer)
print(report)

DebugRetriever remembers the query and nodes from the most recent retrieve(); .debug(answer) runs debug_rag() against them (using each node's get_content()). It's a real BaseRetriever, so it's a drop-in replacement anywhere a LlamaIndex retriever is expected, including RetrieverQueryEngine. See examples/llama_index_integration.py for a runnable end-to-end example.

Requires the llama-index extra: pip install -e ".[llama-index]".

Logging

RAG Debugger uses the standard logging module (logger = logging.getLogger(__name__) per module) and configures no handler of its own — as a library, it stays silent unless the consuming application sets one up.

The CLI does configure a handler for you. Pass -v/--verbose (before the subcommand) to see debug-level diagnostics — which judge/model was resolved, retrieval scoring summaries, and per-case warnings during evaluate — on stderr:

rag-debug -v analyze examples/basic.json

Without -v, only warnings and errors are shown (e.g. a judge API call failing, or a dataset case being skipped). Third-party libraries' own debug logs (httpx, sentence-transformers, etc.) are not included even at -v, so this doesn't drown you in noise.

Error handling

Every judge provider wraps its SDK's API errors (auth failures, rate limits, network issues) into a RuntimeError with a clear message, instead of letting an SDK-specific exception escape uncaught — the CLI's analyze/evaluate commands catch this and print a clean Error: ... line rather than a raw traceback. evaluate_dataset() / rag-debug evaluate treats a per-case runtime failure (e.g. one transient API error in a 50-question dataset) as that one case being skipped, not the whole run failing — DatasetEvaluation.failed lists which cases errored and why, and both the printed summary and any --saved results reflect only the cases that actually completed.

A dataset file with a case missing a required field (question/documents/answer) is a different kind of problem — a dataset-authoring mistake, not a runtime hiccup — and still fails the whole run immediately with a clear error, on the reasoning that silently skipping a malformed case could hide a bug in how the dataset was generated.

Known limitations

• Retrieval relevance uses raw cosine similarity from all-MiniLM-L6-v2. Real-world scores for genuinely relevant pairs often land around 0.3-0.5, not near 1.0 — the default threshold is tuned for this, but may need adjusting for your domain.
• The default Groq judge uses a free-tier model. Even at temperature=0, Groq's batched serving does not guarantee identical output across calls, so on borderline paraphrase cases (e.g. "return" vs "refund") the verdict can occasionally flip between runs. Switching to a larger/stronger model on any provider (--judge openai:... or --judge anthropic:...) generally improves consistency.

Development

pip install -e ".[dev]"

python3 -m pytest tests/ -v      # run the test suite (no API keys/network needed)
ruff check .                     # lint
ruff format .                    # format

See test_strategy.md for what the automated suite covers (and deliberately doesn't), and SELF_TEST.md for the manual checklist to run before publishing a new version.