RAGXRay

RAGXRay is a lightweight RAG debugging toolkit that answers one question:

Why did my RAG application give this answer?

It takes the query, the generated answer, and the retrieved contexts from your RAG pipeline and produces an explainable diagnostic report: scores for retrieval quality, context sufficiency, and answer grounding; a single primary root cause; and concrete recommendations for fixing it.

RAGXRay is not a RAG framework, vector database, LLM wrapper, or dashboard. It doesn't call any LLM, download any embedding model, or require an API key. Everything runs offline using explainable heuristics (TF-IDF similarity, token overlap, and pattern matching for numbers/dates).

Install

pip install -e .

(Once published: pip install ragxray.)

Quickstart

from ragxray import diagnose

report = diagnose(
    query="What is the waiting period for pre-existing diseases?",
    answer="The waiting period is 48 months.",
    contexts=[
        "Pre-existing diseases are covered after 36 months.",
        "Other conditions have a waiting period of 24 months.",
    ],
)

report.show()

RAGXRAY DIAGNOSTIC REPORT

Overall Score: 58/100
Status: Poor

Scores:
Retrieval Quality: 94/100
Context Sufficiency: 100/100
Answer Grounding: 0/100

Primary Root Cause:
UNSUPPORTED_ANSWER
(heuristic confidence: 0.80 - not a statistical probability)

Issues:
- Answer mentions a number not found in the matched context: The waiting period is 48 months.
- The answer is not supported by the retrieved evidence.

Recommendations:
- Add grounding checks and restrict generation to retrieved evidence.
- Return "I don't know" when evidence is insufficient.

What it checks

Root causes

RAGXRay always names exactly one primary root cause:

RETRIEVAL_FAILURE       - retrieval found little/nothing relevant
LOW_RETRIEVAL_RELEVANCE - retrieval is weak but not a total failure
INSUFFICIENT_CONTEXT    - the retrieved contexts don't contain the needed facts
UNSUPPORTED_ANSWER      - the answer states things the context doesn't support
CONTRADICTORY_ANSWER    - the answer directly contradicts the retrieved evidence
UNKNOWN                 - no significant issue detected by current heuristics

Note: confidence on the root cause is a heuristic diagnostic score in [0, 1], not a calibrated statistical probability. It reflects how strongly the detected signals point at the chosen root cause.

Working with the report

report.score               # 0-100 overall score
report.status               # "Excellent" | "Good" | "Needs Attention" | "Poor" | "Critical"
report.retrieval_score
report.context_score
report.grounding_score
report.primary_root_cause   # RootCause enum
report.issues               # list[str] of detected problems
report.recommendations      # list[str] of actionable next steps

report.to_dict()            # plain JSON-serializable dict
report.to_json()            # JSON string
report.show()               # pretty-print to the terminal

Each analyzer's full detail is also available:

report.retrieval_analysis.per_context     # per-context relevance scores
report.context_analysis.missing_terms     # query terms not found anywhere
report.grounding_analysis.unsupported_claims  # specific unsupported/contradicted claims

Context input formats

Plain strings:

contexts = ["Some text", "Another text"]

Structured, with metadata:

contexts = [
    {
        "text": "Pre-existing diseases are covered after 36 months.",
        "source": "policy.pdf",
        "page": 12,
    }
]

Both (and a mix of the two) are normalized internally into a Context model.

Scoring

Overall Score = 30% Retrieval Quality
              + 30% Context Sufficiency
              + 40% Answer Grounding

90-100  Excellent
75-89   Good
60-74   Needs Attention
40-59   Poor
0-39    Critical

All weights and thresholds are configurable via RAGXRayConfig:

from ragxray import diagnose, RAGXRayConfig

config = RAGXRayConfig(
    weight_retrieval=0.2,
    weight_context=0.2,
    weight_grounding=0.6,
    grounding_sentence_support_threshold=0.4,
)

report = diagnose(query=..., answer=..., contexts=..., config=config)

How the heuristics work (v1)

• Retrieval Quality: TF-IDF cosine similarity between the query and each context (falls back to token overlap if TF-IDF's vocabulary is empty), rescaled against a calibrated "strong match" reference point.
• Context Sufficiency: fraction of the query's meaningful (non-stopword) terms that appear anywhere in the retrieved contexts.
• Answer Grounding: each answer sentence is matched against the most similar context sentence. If nothing is similar enough, the claim is unsupported. If something is similar but numbers/dates disagree, that's flagged as unsupported with the specific mismatch. Explicit negation clashes (e.g. "is covered" vs. "is not covered") are flagged as direct contradictions.

These are explainable heuristics, not semantic entailment or fact-checking models - RAGXRay tells you what looks wrong and why it flagged it, but it can't verify real-world truth, and it can miss issues heuristics simply aren't built to catch (e.g. subtle logical errors phrased without numbers or negation words).

What's intentionally NOT in v1

• No citation-level analysis
• No chunk-level analysis
• No LangChain/framework integrations
• No LLM-based evaluation
• No embeddings requiring model downloads

These may come in future versions - v1 is deliberately focused and offline.

Development

pip install -e ".[dev]"
pytest

License

MIT