rag-oracle 0.1.2

RAG Oracle – failure diagnostics for Retrieval-Augmented Generation systems

🔮 RAG-ORACLE

The First Production-Grade Root Cause Analysis System for RAG Failures

RAG-ORACLE is the world's first automated diagnostic system that tells you exactly why your RAG pipeline failed and how to fix it—with production-grade accuracy and zero guesswork.

Add 3 lines of code. Get instant root cause diagnosis.

---

🎯 What Problem Does RAG-ORACLE Solve?

The RAG Debugging Crisis

You built a RAG system. It works... sometimes. When it fails, you're left staring at:

• ❌ Low evaluation scores (faithfulness: 0.3, recall: 0.4)
• ❌ Generic metrics that don't tell you why it failed
• ❌ Hours of manual debugging to find the root cause
• ❌ Trial-and-error fixes that may or may not work

Existing tools like LangSmith and RAGAS give you metrics. RAG-ORACLE gives you answers.

Why LangSmith & RAGAS Fall Short

Feature	LangSmith	RAGAS	RAG-ORACLE
Evaluation Metrics	✅ Yes	✅ Yes	✅ Yes
Root Cause Diagnosis	❌ No	❌ No	✅ YES
Actionable Fixes	❌ Manual	❌ Manual	✅ Automated
Corpus vs Retrieval Separation	❌ No	❌ No	✅ YES
Cost Waste Detection	❌ No	❌ No	✅ YES
Fix Validation	❌ No	❌ No	✅ YES
Hallucination Detection	⚠️ Basic	⚠️ Basic	✅ Advanced

The Problem: LangSmith and RAGAS tell you "faithfulness is low". They don't tell you if it's because:

• Your corpus is missing information (unfixable without new data)
• Your retrieval config is wrong (fixable by increasing top_k)
• Your embedding model doesn't understand your domain (fixable by switching models)
• Your LLM is hallucinating (fixable by lowering temperature)

RAG-ORACLE solves this. It's the difference between a thermometer (tells you there's a fever) and a doctor (tells you it's strep throat and prescribes antibiotics).

---

🏗️ System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         RAG PIPELINE                             │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐    │
│  │  Query   │──▶│ Retrieval│──▶│Generation│──▶│  Answer  │    │
│  └──────────┘   └──────────┘   └──────────┘   └──────────┘    │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    RAG-ORACLE DIAGNOSTIC ENGINE                  │
│                                                                   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 1: Signal Collection                               │   │
│  │  • Evaluation Metrics (Faithfulness, Recall, Relevance)  │   │
│  │  • Query Feasibility (Typos, Constraints)                │   │
│  │  • Corpus Concept Check (Missing vs Available)           │   │
│  │  • Cost Analysis (Token Usage, Waste)                    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 2: Root Cause Analysis (10 Diagnostic Rules)       │   │
│  │  1. User Query Issues (Typos, Grammar)                   │   │
│  │  2. Corpus Coverage (Missing Information)                │   │
│  │  3. Retrieval Configuration (top_k, chunk_size)          │   │
│  │  4. Retrieval Noise (Irrelevant chunks)                  │   │
│  │  5. Embedding Mismatch (Domain-specific)                 │   │
│  │  6. Prompt Design (Task alignment)                       │   │
│  │  7. Hallucination Risk (Ungrounded claims)               │   │
│  │  8. Generation Control (Temperature)                     │   │
│  │  9. Cost Inefficiency (Wasted tokens)                    │   │
│  │ 10. Systemic Drift (Performance degradation)             │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 3: Fix Recommendation & Validation                 │   │
│  │  • Actionable Fix (e.g., "Increase top_k from 3 to 5")  │   │
│  │  • Confidence Score (0.0 - 1.0)                          │   │
│  │  • Fix Validation (Before/After comparison)              │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Output: Diagnosis Report                                 │   │
│  │  • Primary Failure Type                                  │   │
│  │  • User-Friendly Explanation                             │   │
│  │  • Recommended Fix                                       │   │
│  │  • Is Unfixable? (Corpus vs System)                      │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

---

📊 Real-World Example

Input Query

question = "What is the author's view on fear of failure?"

RAG-ORACLE Output

{
  "query_id": "abc123",
  "outcome": "FAILURE",
  "primary_failure": "Corpus Coverage",
  "recommended_fix": "Expand corpus. Missing: failure, fear, psychology.",
  "is_unfixable": true,
  "confidence": 0.82,
  "explanation": "Your system failed because the required information does not exist in your documents. No retrieval or prompt tuning can fix this. You must add documents covering: failure, fear, psychology.",
  "diagnostic_maturity": "high-confidence"
}

What This Means

• ✅ Clear Diagnosis: The information doesn't exist in your corpus
• ✅ Actionable: Add documents about "failure", "fear", "psychology"
• ✅ Honest: Marked as is_unfixable because no system tweak will help
• ✅ Confident: 82% confidence in this diagnosis

---

🏥 System Health Report Example

After running 50 queries, get a comprehensive health report:

{
  "system_verdict": "Retrieval-Constrained System. Adjust retrieval parameters.",
  "total_queries": 50,
  "failed_queries": 15,
  "success_queries": 30,
  "success_with_risk": 5,
  "failure_rate": 0.30,
  "most_common_failure": {
    "type": "Retrieval Configuration",
    "count": 8,
    "percentage": 53.3
  },
  "immediate_action": "Increase top_k from 3 to 5",
  "strategic_action": "Increase top_k from 3 to 5 or adjust chunk_size",
  "total_cost_waste_usd": 0.002340,
  "total_cost_saved_usd": 0.001200,
  "average_confidence": 0.76
}

Insights:

• 30% failure rate, mostly due to retrieval config
• Immediate fix: Increase top_k from 3 to 5
• Saved $0.0012 by optimizing successful queries
• High confidence (0.76) in diagnoses

---

🚀 Quickstart

Installation

pip install rag-oracle

Prerequisites

RAG-ORACLE works with any RAG system. You'll need:

• Python 3.8+
• Your existing RAG pipeline (OpenAI, LangChain, LlamaIndex, or custom)
• Optional: Embedding model for auto-evaluation (recommended for easiest setup)

Setting Up Embeddings (Recommended)

For auto-evaluation, you need an embedding model. Here are common options:

# Option 1: HuggingFace (Free, no API key needed)
from langchain_huggingface import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

# Option 2: OpenAI (Requires API key)
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(openai_api_key="your-key")

# Option 3: Use your existing embeddings
# Any LangChain-compatible embedding model works

Note: If you don't provide embeddings, you must provide evaluation metrics manually in each diagnose() call (see Advanced Usage).

Minimal Usage (3 Lines)

For existing RAG systems - super simple integration:

from rag_pipeline import RAGOracle
from langchain_huggingface import HuggingFaceEmbeddings

# Initialize (one time setup)
# Option 1: With embeddings for auto-evaluation (recommended)
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
oracle = RAGOracle(embeddings=embeddings)

# Option 2: Without embeddings (you provide evaluation manually)
# oracle = RAGOracle()

# In your existing RAG code, add this:
result = oracle.diagnose(
    query="What is the capital of France?",
    answer=your_rag_answer,  # Your RAG system's generated answer
    chunks=your_retrieved_chunks,  # List of retrieved document chunks
    config={"top_k": 5, "temperature": 0.7}  # Optional: your system config
)

# Get instant diagnosis
if result["root_causes"]:
    primary = result["root_causes"][0]
    print(f"💡 Fix: {primary['fix']}")
    print(f"📝 {primary['user_explanation']}")
    print(f"⚠️ Unfixable: {primary['is_unfixable']}")

That's it. No pipeline setup. No configuration. Just diagnose.

Note: If you don't provide embeddings, you must provide evaluation in each diagnose() call (see Advanced Usage below).

Full Pipeline Usage

If you want a complete RAG pipeline with built-in diagnostics:

from rag_pipeline import RAGPipeline

# Initialize with your documents
pipeline = RAGPipeline(document_source="./documents")

# Query and get diagnosis
result = pipeline.query("What is the capital of France?")

# Get root cause analysis
print(result["root_causes"])

Get System Health Report

# Get aggregated insights across all queries
report = oracle.get_report()

print(f"Failure Rate: {report['failure_rate']*100:.1f}%")
print(f"Most Common Issue: {report['most_common_failure']['type']}")
print(f"Immediate Action: {report['immediate_action']}")
# Output:
# Failure Rate: 30.0%
# Most Common Issue: Retrieval Configuration
# Immediate Action: Increase top_k from 3 to 5

# Analyze only last 10 queries
recent_report = oracle.get_report(last_n=10)

Public Output Example

# Get clean, user-facing output (for APIs/production)
# Pass the result from diagnose() method
public_output = oracle.get_public_output(result)

print(public_output)
# {
#   "query_id": "xyz789",
#   "outcome": "SUCCESS_WITH_RISK",
#   "primary_failure": "Cost Inefficiency",
#   "recommended_fix": "Reduce top_k from 5",
#   "is_unfixable": false,
#   "confidence": 0.68,
#   "explanation": "Your system is retrieving more chunks than necessary...",
#   "diagnostic_maturity": "stable"
# }

Fix Validation Example

# After applying a fix, validate if it worked
validation = oracle.validate_fix(
    before_query_ids=["query_1", "query_2", "query_3"],  # Before fix
    after_query_ids=["query_4", "query_5", "query_6"]    # After fix
)

print(f"Fix Applied: {validation['fix_applied']}")
print(f"Failure Rate Change: {validation['failure_rate_change']}")
print(f"Verdict: {validation['verdict']}")
# Output:
# Fix Applied: Increase top_k from 3 to 5
# Failure Rate Change: -20%
# Verdict: Fix effective

---

📚 Basic Usage

Complete Example

from rag_pipeline import RAGPipeline
import json

# 1. Initialize pipeline
pipeline = RAGPipeline(
    document_source="./documents",
    chunk_size=500,
    chunk_overlap=50,
    embedding_model="huggingface",
    top_k=3,
    temperature=0.7
)

# 2. Query the system
question = "What are the main themes in the document?"
result = pipeline.query(question)

# 3. View the answer
print(f"Answer: {result['answer']}")

# 4. Check for failures
root_causes = result.get("root_causes", [])
if root_causes:
    primary = root_causes[0]
    print(f"\n⚠️ Issue Detected: {primary['type']}")
    print(f"Fix: {primary['fix']}")
    print(f"Explanation: {primary['user_explanation']}")
    print(f"Confidence: {primary['confidence']:.2f}")

# 5. Get system health report
report = pipeline.root_cause_oracle.get_report(last_n=10)
print(json.dumps(report, indent=2))

---

💼 Common Use Cases

Use Case 1: Quick Diagnosis for Single Query

from rag_pipeline import RAGOracle
from langchain_huggingface import HuggingFaceEmbeddings

oracle = RAGOracle(embeddings=HuggingFaceEmbeddings())

# After your RAG system generates an answer
result = oracle.diagnose(
    query="What is the capital of France?",
    answer=rag_answer,
    chunks=retrieved_chunks
)

# Check if there are issues
if result["root_causes"]:
    print(f"Issue: {result['root_causes'][0]['type']}")
    print(f"Fix: {result['root_causes'][0]['fix']}")

Use Case 2: Batch Processing with Health Reports

# Process multiple queries
queries = ["query1", "query2", "query3"]
results = []

for query in queries:
    answer, chunks = your_rag_system.process(query)
    result = oracle.diagnose(query=query, answer=answer, chunks=chunks)
    results.append(result)

# Get system-wide health report
health_report = oracle.get_report()
print(f"Overall Failure Rate: {health_report['failure_rate']*100:.1f}%")
print(f"Most Common Issue: {health_report['most_common_failure']['type']}")

Use Case 3: Fix Validation Workflow

# Step 1: Collect queries before fix
before_queries = []
for query in test_queries:
    result = oracle.diagnose(...)
    before_queries.append(result["query_id"])

# Step 2: Apply recommended fix (e.g., increase top_k from 3 to 5)
your_rag_system.update_config(top_k=5)

# Step 3: Collect queries after fix
after_queries = []
for query in test_queries:
    result = oracle.diagnose(...)
    after_queries.append(result["query_id"])

# Step 4: Validate if fix worked
validation = oracle.validate_fix(
    before_query_ids=before_queries,
    after_query_ids=after_queries
)

if validation["verdict"] == "Fix effective":
    print("✅ Fix validated! Keep the change.")
else:
    print("❌ Fix didn't help. Try a different approach.")

Use Case 4: Production API Integration

from flask import Flask, request, jsonify
from rag_pipeline import RAGOracle

app = Flask(__name__)
oracle = RAGOracle(embeddings=your_embeddings)

@app.route("/diagnose", methods=["POST"])
def diagnose_endpoint():
    data = request.json
    result = oracle.diagnose(
        query=data["query"],
        answer=data["answer"],
        chunks=data["chunks"]
    )
    
    # Return clean public output
    return jsonify(oracle.get_public_output(result))

🔧 Integration Guide

Option 1: Simple Integration (Recommended)

For 99% of users - just 3 lines of code:

from rag_pipeline import RAGOracle
from langchain_huggingface import HuggingFaceEmbeddings

# Initialize once (with your embeddings for auto-evaluation)
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
oracle = RAGOracle(embeddings=embeddings)

# In your existing RAG code, add this:
result = oracle.diagnose(
    query=user_query,
    answer=your_generated_answer,
    chunks=your_retrieved_chunks,  # Can be strings, dicts, or LangChain Documents
    config={"top_k": 5, "temperature": 0.7}  # optional
)

# Get actionable fix
if result["root_causes"]:
    fix = result["root_causes"][0]["fix"]
    explanation = result["root_causes"][0]["user_explanation"]
    is_unfixable = result["root_causes"][0]["is_unfixable"]
    print(f"💡 Fix: {fix}")
    print(f"📝 {explanation}")
    if is_unfixable:
        print("⚠️ This requires data changes, not system tuning.")

Works with any RAG system - OpenAI, LangChain, LlamaIndex, custom pipelines.

Chunk Format Support: The chunks parameter accepts:

• List of strings: ["chunk 1 text", "chunk 2 text"]
• List of dicts: [{"page_content": "text", "metadata": {...}}, ...]
• List of LangChain Documents: [Document(page_content="..."), ...]

Option 2: Advanced Integration (Manual Evaluation)

For custom evaluation pipelines or when you already have evaluation metrics:

from rag_pipeline import RAGOracle

# Initialize without embeddings (you'll provide evaluation manually)
oracle = RAGOracle()

# Your existing RAG pipeline
question = "Your query"
chunks = your_retriever.retrieve(question)
answer = your_generator.generate(question, chunks)

# Your custom evaluation (must include faithfulness, relevance, context_recall)
evaluation = {
    "faithfulness": {
        "faithfulness": 0.65,  # Score 0.0-1.0
        "unsupported_claims": ["claim 1", "claim 2"]
    },
    "relevance": {
        "relevance": 0.72
    },
    "context_recall": {
        "context_recall": 0.58,
        "missing_concepts": ["concept1", "concept2"]
    }
}

# Diagnose with pre-computed evaluation
result = oracle.diagnose(
    query=question,
    answer=answer,
    chunks=chunks,
    evaluation=evaluation,  # Provide your evaluation
    config={"top_k": 3, "temperature": 0.7}
)

print(result["root_causes"])

Option 3: Direct RootCauseOracle Usage

For maximum control and custom signal processing:

from rag_pipeline import RootCauseOracle

# Initialize Oracle
oracle = RootCauseOracle(query_history_file="./query_history.json")

# Prepare signals dictionary
signals = {
    "query_id": "unique_query_id",
    "question": "Your query",
    "answer": "Generated answer",
    "retrieved_chunks": chunks,  # List of LangChain Documents
    "evaluation": {
        "faithfulness": {"faithfulness": 0.65, "unsupported_claims": []},
        "relevance": {"relevance": 0.72},
        "context_recall": {"context_recall": 0.58, "missing_concepts": []}
    },
    "config": {"top_k": 3, "temperature": 0.7},
    "query_feasibility": None,  # Optional
    "cost_optimization": None,  # Optional
    "corpus_concept_check": None  # Optional
}

# Analyze
diagnosis = oracle.analyze(signals)
print(diagnosis["root_causes"])

---

📖 API Reference

RAGPipeline

Main entry point for the RAG system with built-in diagnostics.

Constructor

RAGPipeline(
    document_source: str,           # Path to documents
    chunk_size: int = 500,          # Chunk size in characters
    chunk_overlap: int = 50,        # Overlap between chunks
    embedding_model: str = "huggingface",  # Embedding model
    top_k: int = 3,                 # Number of chunks to retrieve
    temperature: float = 0.7,       # Generation temperature
    enable_evaluation: bool = True  # Enable diagnostics
)

Methods

query(question: str) -> dict

• Query the RAG system and get diagnosis
• Returns: Full result with answer, evaluation, and root causes

get_config() -> dict

• Get current pipeline configuration

update_top_k(top_k: int)

• Update retrieval parameter

update_temperature(temperature: float)

• Update generation parameter

---

RAGOracle

High-level wrapper for simplified integration (recommended for most users).

Constructor

RAGOracle(
    query_history_file: Optional[str] = None,  # Defaults to "./query_history.json"
    embeddings: Optional[Any] = None,          # Embedding model for auto-evaluation
    generator: Optional[Any] = None            # Generator for LLM-as-a-Judge evaluation
)

Methods

diagnose(query: str, answer: str, chunks: list, config: dict = None, evaluation: dict = None, query_id: str = None) -> dict

• Diagnose RAG query execution and identify root causes
• query: User's question/query string
• answer: Generated answer from your RAG system
• chunks: Retrieved document chunks. Can be:
  • List of strings: ["chunk text 1", "chunk text 2"]
  • List of dicts: [{"page_content": "text", "metadata": {...}}, ...]
  • List of LangChain Documents: [Document(...), ...]
• config: Optional dict with system config (e.g., {"top_k": 5, "temperature": 0.7})
• evaluation: Optional pre-computed evaluation dict. If not provided and embeddings available, will auto-evaluate
• query_id: Optional query identifier (auto-generated if not provided)
• Returns: Diagnosis dict with root_causes, primary_failure, outcome, etc.

get_report(last_n: int = None) -> dict

• Get system health report
• last_n: Analyze last N queries (None = all)

validate_fix(fix_id: str = None, before_query_ids: list = None, after_query_ids: list = None) -> dict

• Validate if a fix actually worked
• Either provide fix_id OR before_query_ids/after_query_ids
• Returns: Before/after comparison with verdict

apply_fix(fix_id: str) -> bool

• Mark a recommended fix as "applied" in fix history
• Returns: True if fix was found and marked, False otherwise

get_public_output(result: dict) -> dict

• Get clean, user-facing output
• Returns: Simplified diagnosis for APIs/production

---

RootCauseOracle

Core diagnostic engine for root cause analysis.

Constructor

RootCauseOracle(
    query_history_file: str = "./query_history.json"
)

Methods

analyze(signals: dict) -> dict

• Analyze query execution signals
• Returns: Diagnosis with root causes and fixes

get_report(last_n: int = None) -> dict

• Get system health report
• last_n: Analyze last N queries (None = all)

validate_fix(fix_id: str = None, before_query_ids: list = None, after_query_ids: list = None) -> dict

• Validate if a fix actually worked
• Either provide fix_id OR before_query_ids/after_query_ids
• Returns: Before/after comparison with verdict

apply_fix(fix_id: str) -> bool

• Mark a recommended fix as "applied" in fix history
• Returns: True if fix was found and marked, False otherwise

get_public_output(result: dict) -> dict

• Get clean, user-facing output
• Returns: Simplified diagnosis

---

⚙️ Configuration Options

Pipeline Configuration

config = {
    # Chunking
    "chunk_size": 500,        # Characters per chunk
    "chunk_overlap": 50,      # Overlap between chunks
    
    # Retrieval
    "top_k": 3,               # Number of chunks to retrieve
    "embedding_model": "huggingface",  # or "openai"
    
    # Generation
    "temperature": 0.7,       # 0.0 (deterministic) to 1.0 (creative)
    "model_type": "openai",   # or "groq"
    "model_name": "gpt-4",    # Specific model
    
    # Diagnostics
    "enable_evaluation": True,
    "enable_tracing": True,
    "log_dir": "./logs"
}

pipeline = RAGPipeline(**config)

Environment Variables

# API Keys
OPENAI_API_KEY=your_openai_key
GROQ_API_KEY=your_groq_key

# Optional
RAG_LOG_DIR=./logs
RAG_VECTOR_STORE=./vector_store

---

🎓 Core Concepts

1. Corpus Coverage vs Retrieval Failure

The Critical Distinction

Scenario	Corpus Coverage	Retrieval Failure
Problem	Info doesn't exist in docs	Info exists but not retrieved
Cause	Missing documents	Wrong top_k, bad embeddings
Fix	Add documents (unfixable by system)	Increase top_k (fixable)
Example	"What's the CEO's salary?" (not in docs)	"What's the company mission?" (in docs, not retrieved)

How RAG-ORACLE Detects This:

# Checks if missing concepts exist in corpus
corpus_check = retriever.check_concepts_in_corpus(missing_concepts)

if concepts_missing_from_corpus:
    # Corpus Coverage - unfixable
    diagnosis = "Expand corpus. Missing: [concepts]"
    is_unfixable = True
else:
    # Retrieval Failure - fixable
    diagnosis = "Increase top_k from 3 to 5"
    is_unfixable = False

---

2. Correct Abstention

What is it? When the RAG system correctly says "I don't know" because the information isn't in the corpus.

Why it matters?

• ✅ Good: System abstains when info is missing (honest)
• ❌ Bad: System hallucinates an answer (dangerous)

How RAG-ORACLE Handles It:

# Detects abstention language
abstention_phrases = [
    "does not contain enough information",
    "cannot be answered from the provided context",
    "not found in the context"
]

if abstention_detected and info_missing_from_corpus:
    # This is GOOD behavior - don't flag as failure
    outcome = "SUCCESS_WITH_RISK"
    explanation = "System correctly abstained. Consider expanding corpus."

Example:

Q: "What is the CEO's favorite color?"
A: "The provided context does not contain information about the CEO's favorite color."

Diagnosis: ✅ Correct Abstention (not a failure!)

---

3. Fix Validation

The Problem: You applied a fix. Did it actually work?

RAG-ORACLE's Solution: Automated before/after comparison

# Before fix
before_queries = ["query1", "query2", "query3"]
# Apply fix: top_k = 3 → 5
# After fix
after_queries = ["query4", "query5", "query6"]

# Validate
validation = oracle.validate_fix(before_query_ids=before_queries, after_query_ids=after_queries)

print(validation)
# {
#   "fix_applied": "Increase top_k from 3 to 5",
#   "failure_rate_change": "-20%",  # 20% reduction in failures
#   "retrieval_recall_change": "+0.15",  # Recall improved by 0.15
#   "verdict": "Fix effective"
# }

Validation Criteria:

• ✅ Effective: Failure rate drops >5% OR recall improves >0.05
• ❌ Ineffective: Failure rate increases OR recall drops
• ⚠️ No Change: Metrics stay within ±5%

---

4. Cost Awareness

The Hidden Problem: Your RAG system works, but wastes money.

How RAG-ORACLE Detects Waste:

# Scenario: Retrieved 10 chunks, only used 2
cost_analysis = {
    "total_tokens": 5000,
    "wasted_tokens": 4000,  # 8 unused chunks
    "wasted_cost_usd": 0.0008,
    "unused_chunks_ratio": 0.8
}

# Diagnosis
if unused_ratio >= 0.4:
    diagnosis = "Cost Inefficiency"
    fix = "Reduce top_k from 10 to 3"
    explanation = "You're retrieving 10 chunks but only using 2. Reduce top_k to save costs."

Cost Optimization Triggers:

1. Unused Chunks: Retrieved 10, used 2
2. High Cost Per Token: Paying too much for little value
3. Metadata Overfetch: Retrieved 5 sources, used 1

Real Impact:

• Before: 10 chunks × 500 tokens = 5000 tokens
• After: 3 chunks × 500 tokens = 1500 tokens
• Savings: 70% token reduction

---

🔍 Root Cause Types

1. User Query Issues

• Typos: "What is the captial of France?" → "capital"
• Ambiguous Grammar: "The fear of failure" (statement, not question)
• Over-Constrained: "List exactly 5 items" (too rigid)

Fix: Query preprocessing, normalization

---

2. Corpus Coverage

• Missing Information: Required data not in documents
• Unfixable: System can't fix this—user must add documents

Fix: Expand corpus with relevant documents

---

3. Retrieval Configuration

• Low Recall: Info exists but not retrieved
• Cause: top_k too low, chunks too small

Fix: Increase top_k, adjust chunk_size

---

4. Retrieval Noise

• High Recall, Low Relevance: Retrieved too much junk
• Cause: top_k too high, poor filtering

Fix: Reduce top_k, improve chunk filtering

---

5. Embedding Mismatch

• Domain-Specific Failure: Generic embeddings don't understand domain jargon
• Cause: Using general-purpose embeddings for specialized domain

Fix: Switch to domain-specific embeddings or fine-tune

---

6. Prompt Design

• Good Retrieval, Bad Answer: Retrieved right info, generated wrong answer
• Cause: Prompt doesn't align with task

Fix: Rewrite system prompt

---

7. Hallucination Risk

• Ungrounded Claims: Answer contains facts not in retrieved text
• Cause: High temperature, creative generation

Fix: Lower temperature, enforce extractive answering

---

8. Generation Control

• Unsupported Claims: LLM making things up
• Cause: Temperature too high

Fix: Lower temperature from 0.7 to 0.3

---

9. Cost Inefficiency

• Wasted Tokens: Retrieving more than needed
• Cause: top_k too high for simple queries

Fix: Reduce top_k, compress context

---

10. Systemic Drift

• Performance Degradation: System worked before, failing now
• Cause: Config changes, corpus updates, model changes

Fix: Review recent changes, rollback if needed

---

🆚 RAG-ORACLE vs Alternatives

Tool	What It Does	Integration	RAG-ORACLE Advantage
LangSmith	Tracing & monitoring (what happened)	Complex setup	✅ 3-line integration + tells you WHY
RAGAS	Evaluation metrics (scores)	Requires pipeline	✅ Works with any RAG + actionable fixes
Manual Debugging	Trial-and-error	N/A	✅ Instant diagnosis instead of hours

The Key Difference:

• LangSmith/RAGAS: "Your faithfulness score is 0.3" ❓
• RAG-ORACLE: "Your faithfulness is low because corpus is missing info. Add documents about X, Y, Z." ✅

Use Together: LangSmith for tracing + RAGAS for metrics + RAG-ORACLE for diagnosis = Complete RAG observability

---

🔍 Troubleshooting

Common Issues

1. "Evaluation not provided and no embeddings available"

# Solution: Either provide embeddings during initialization
oracle = RAGOracle(embeddings=your_embeddings)

# OR provide evaluation in each diagnose() call
result = oracle.diagnose(..., evaluation=your_evaluation)

2. "Unsupported chunk type"

# Solution: Ensure chunks are in one of these formats:
chunks = ["text1", "text2"]  # List of strings
chunks = [{"page_content": "text", "metadata": {}}]  # List of dicts
chunks = [Document(page_content="text")]  # List of LangChain Documents

3. Import Error: "No module named 'rag_pipeline'"

# Solution: Install the package
pip install rag-oracle

# Or install from source
pip install -e .

4. Low confidence scores in diagnoses

• This is normal for new systems with limited query history
• Confidence improves as the system learns from more queries
• Use get_report() to see system-wide patterns

Getting Help

• Check the examples/ directory for working code
• Review the API Reference section above
• Open an issue on GitHub

📦 Package Structure

rag-oracle/
├── rag_pipeline/
│   ├── root_cause_oracle.py    # Core diagnostic engine
│   ├── rag_oracle.py           # Simplified wrapper (recommended)
│   ├── pipeline.py             # Full RAG pipeline (optional)
│   ├── evaluator.py            # Evaluation metrics
│   ├── config.py               # Configuration defaults
│   └── ...                     # Other components
├── examples/
│   └── example_usage.py        # Usage examples
├── README.md                    # This file
├── LICENSE                      # MIT License
└── pyproject.toml              # Package configuration

---

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

---

📄 License

MIT License - see LICENSE for details.

---

🙏 Acknowledgments

Built with:

• LangChain for RAG components
• OpenAI/Groq for LLMs
• HuggingFace for embeddings

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Repository: RAG-ORACLE on GitHub

---

🎯 Roadmap

• Support for more LLM providers (Anthropic, Cohere)
• Web UI for diagnostics dashboard
• Automated fix application
• Multi-language support
• Enterprise features (team collaboration, audit logs)

---

Made with MUHAMMAD HASSAN SHAHBAZ by developers who are tired of debugging RAG systems manually.

RAG-ORACLE: Because your RAG system deserves a doctor, not just a thermometer.