rag-oracle 0.1.0

RAG Oracle – failure diagnostics for Retrieval-Augmented Generation systems

🔮 RAG-ORACLE

The First Production-Grade Root Cause Analysis System for RAG Failures

RAG-ORACLE is the world's first automated diagnostic system that tells you exactly why your RAG pipeline failed and how to fix it—with production-grade accuracy and zero guesswork.

Add 3 lines of code. Get instant root cause diagnosis.

---

🎯 What Problem Does RAG-ORACLE Solve?

The RAG Debugging Crisis

You built a RAG system. It works... sometimes. When it fails, you're left staring at:

• ❌ Low evaluation scores (faithfulness: 0.3, recall: 0.4)
• ❌ Generic metrics that don't tell you why it failed
• ❌ Hours of manual debugging to find the root cause
• ❌ Trial-and-error fixes that may or may not work

Existing tools like LangSmith and RAGAS give you metrics. RAG-ORACLE gives you answers.

Why LangSmith & RAGAS Fall Short

Feature	LangSmith	RAGAS	RAG-ORACLE
Evaluation Metrics	✅ Yes	✅ Yes	✅ Yes
Root Cause Diagnosis	❌ No	❌ No	✅ YES
Actionable Fixes	❌ Manual	❌ Manual	✅ Automated
Corpus vs Retrieval Separation	❌ No	❌ No	✅ YES
Cost Waste Detection	❌ No	❌ No	✅ YES
Fix Validation	❌ No	❌ No	✅ YES
Hallucination Detection	⚠️ Basic	⚠️ Basic	✅ Advanced

The Problem: LangSmith and RAGAS tell you "faithfulness is low". They don't tell you if it's because:

• Your corpus is missing information (unfixable without new data)
• Your retrieval config is wrong (fixable by increasing top_k)
• Your embedding model doesn't understand your domain (fixable by switching models)
• Your LLM is hallucinating (fixable by lowering temperature)

RAG-ORACLE solves this. It's the difference between a thermometer (tells you there's a fever) and a doctor (tells you it's strep throat and prescribes antibiotics).

---

🏗️ System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         RAG PIPELINE                             │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐    │
│  │  Query   │──▶│ Retrieval│──▶│Generation│──▶│  Answer  │    │
│  └──────────┘   └──────────┘   └──────────┘   └──────────┘    │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    RAG-ORACLE DIAGNOSTIC ENGINE                  │
│                                                                   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 1: Signal Collection                               │   │
│  │  • Evaluation Metrics (Faithfulness, Recall, Relevance)  │   │
│  │  • Query Feasibility (Typos, Constraints)                │   │
│  │  • Corpus Concept Check (Missing vs Available)           │   │
│  │  • Cost Analysis (Token Usage, Waste)                    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 2: Root Cause Analysis (10 Diagnostic Rules)       │   │
│  │  1. User Query Issues (Typos, Grammar)                   │   │
│  │  2. Corpus Coverage (Missing Information)                │   │
│  │  3. Retrieval Configuration (top_k, chunk_size)          │   │
│  │  4. Retrieval Noise (Irrelevant chunks)                  │   │
│  │  5. Embedding Mismatch (Domain-specific)                 │   │
│  │  6. Prompt Design (Task alignment)                       │   │
│  │  7. Hallucination Risk (Ungrounded claims)               │   │
│  │  8. Generation Control (Temperature)                     │   │
│  │  9. Cost Inefficiency (Wasted tokens)                    │   │
│  │ 10. Systemic Drift (Performance degradation)             │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Phase 3: Fix Recommendation & Validation                 │   │
│  │  • Actionable Fix (e.g., "Increase top_k from 3 to 5")  │   │
│  │  • Confidence Score (0.0 - 1.0)                          │   │
│  │  • Fix Validation (Before/After comparison)              │   │
│  └─────────────────────────────────────────────────────────┘   │
│                         │                                         │
│                         ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Output: Diagnosis Report                                 │   │
│  │  • Primary Failure Type                                  │   │
│  │  • User-Friendly Explanation                             │   │
│  │  • Recommended Fix                                       │   │
│  │  • Is Unfixable? (Corpus vs System)                      │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

---

📊 Real-World Example

Input Query

question = "What is the author's view on fear of failure?"

RAG-ORACLE Output

{
  "query_id": "abc123",
  "outcome": "FAILURE",
  "primary_failure": "Corpus Coverage",
  "recommended_fix": "Expand corpus. Missing: failure, fear, psychology.",
  "is_unfixable": true,
  "confidence": 0.82,
  "explanation": "Your system failed because the required information does not exist in your documents. No retrieval or prompt tuning can fix this. You must add documents covering: failure, fear, psychology.",
  "diagnostic_maturity": "high-confidence"
}

What This Means

• ✅ Clear Diagnosis: The information doesn't exist in your corpus
• ✅ Actionable: Add documents about "failure", "fear", "psychology"
• ✅ Honest: Marked as is_unfixable because no system tweak will help
• ✅ Confident: 82% confidence in this diagnosis

---

🏥 System Health Report Example

After running 50 queries, get a comprehensive health report:

{
  "system_verdict": "Retrieval-Constrained System. Adjust retrieval parameters.",
  "total_queries": 50,
  "failed_queries": 15,
  "success_queries": 30,
  "success_with_risk": 5,
  "failure_rate": 0.30,
  "most_common_failure": {
    "type": "Retrieval Configuration",
    "count": 8,
    "percentage": 53.3
  },
  "immediate_action": "Increase top_k from 3 to 5",
  "strategic_action": "Increase top_k from 3 to 5 or adjust chunk_size",
  "total_cost_waste_usd": 0.002340,
  "total_cost_saved_usd": 0.001200,
  "average_confidence": 0.76
}

Insights:

• 30% failure rate, mostly due to retrieval config
• Immediate fix: Increase top_k from 3 to 5
• Saved $0.0012 by optimizing successful queries
• High confidence (0.76) in diagnoses

---

🚀 Quickstart

Installation

pip install rag-oracle

Minimal Usage (3 Lines)

For existing RAG systems - super simple integration:

from rag_oracle import RAGOracle

# Initialize (one time setup)
oracle = RAGOracle(embeddings=your_embeddings)  # or use your existing embeddings

# Diagnose any query (add this to your existing code)
result = oracle.diagnose(
    query="What is the capital of France?",
    answer=your_rag_answer,
    chunks=your_retrieved_chunks
)

# Get instant diagnosis
print(result["root_causes"][0]["fix"])  # "Increase top_k from 3 to 5"

That's it. No pipeline setup. No configuration. Just diagnose.

Full Pipeline Usage

If you want a complete RAG pipeline with built-in diagnostics:

from rag_oracle import RAGPipeline

# Initialize with your documents
pipeline = RAGPipeline(document_source="./documents")

# Query and get diagnosis
result = pipeline.query("What is the capital of France?")

# Get root cause analysis
print(result["root_causes"])

Get System Health Report

# Get aggregated insights across all queries
report = oracle.get_report()

print(f"Failure Rate: {report['failure_rate']*100:.1f}%")
print(f"Most Common Issue: {report['most_common_failure']['type']}")
print(f"Immediate Action: {report['immediate_action']}")
# Output:
# Failure Rate: 30.0%
# Most Common Issue: Retrieval Configuration
# Immediate Action: Increase top_k from 3 to 5

Public Output Example

# Get clean, user-facing output (for APIs/production)
public_output = oracle.get_public_output(result)

print(public_output)
# {
#   "query_id": "xyz789",
#   "outcome": "SUCCESS_WITH_RISK",
#   "primary_failure": "Cost Inefficiency",
#   "recommended_fix": "Reduce top_k from 5",
#   "is_unfixable": false,
#   "confidence": 0.68,
#   "explanation": "Your system is retrieving more chunks than necessary...",
#   "diagnostic_maturity": "stable"
# }

---

📚 Basic Usage

Complete Example

from rag_oracle import RAGPipeline
import json

# 1. Initialize pipeline
pipeline = RAGPipeline(
    document_source="./documents",
    chunk_size=500,
    chunk_overlap=50,
    embedding_model="huggingface",
    top_k=3,
    temperature=0.7
)

# 2. Query the system
question = "What are the main themes in the document?"
result = pipeline.query(question)

# 3. View the answer
print(f"Answer: {result['answer']}")

# 4. Check for failures
root_causes = result.get("root_causes", [])
if root_causes:
    primary = root_causes[0]
    print(f"\n⚠️ Issue Detected: {primary['type']}")
    print(f"Fix: {primary['fix']}")
    print(f"Explanation: {primary['user_explanation']}")
    print(f"Confidence: {primary['confidence']:.2f}")

# 5. Get system health report
report = pipeline.root_cause_oracle.get_report(last_n=10)
print(json.dumps(report, indent=2))

---

🔧 Integration Guide

Option 1: Simple Integration (Recommended)

For 99% of users - just 3 lines of code:

from rag_oracle import RAGOracle

# Initialize once (with your embeddings for auto-evaluation)
oracle = RAGOracle(embeddings=your_embeddings)

# In your existing RAG code, add this:
result = oracle.diagnose(
    query=user_query,
    answer=your_generated_answer,
    chunks=your_retrieved_chunks,
    config={"top_k": 5, "temperature": 0.7}  # optional
)

# Get actionable fix
fix = result["root_causes"][0]["fix"]
explanation = result["root_causes"][0]["user_explanation"]
print(f"💡 Fix: {fix}")
print(f"📝 {explanation}")

Works with any RAG system - OpenAI, LangChain, LlamaIndex, custom pipelines.

Option 2: Advanced Integration

For custom evaluation pipelines or advanced use cases:

from rag_oracle import RootCauseOracle

# Your existing RAG components
retriever = YourRetriever()
generator = YourGenerator()
evaluator = YourEvaluator()

# Initialize Oracle
oracle = RootCauseOracle(query_history_file="./logs/query_history.json")

# Run your RAG pipeline
question = "Your query"
chunks = retriever.retrieve(question)
answer = generator.generate(question, chunks)
evaluation = evaluator.evaluate(question, answer, chunks)

# Diagnose with Oracle
signals = {
    "query_id": "unique_id",
    "question": question,
    "answer": answer,
    "retrieved_chunks": chunks,
    "evaluation": evaluation,
    "config": {"top_k": 3, "temperature": 0.7}
}

diagnosis = oracle.analyze(signals)
print(diagnosis["root_causes"])

---

📖 API Reference

RAGPipeline

Main entry point for the RAG system with built-in diagnostics.

Constructor

RAGPipeline(
    document_source: str,           # Path to documents
    chunk_size: int = 500,          # Chunk size in characters
    chunk_overlap: int = 50,        # Overlap between chunks
    embedding_model: str = "huggingface",  # Embedding model
    top_k: int = 3,                 # Number of chunks to retrieve
    temperature: float = 0.7,       # Generation temperature
    enable_evaluation: bool = True  # Enable diagnostics
)

Methods

query(question: str) -> dict

• Query the RAG system and get diagnosis
• Returns: Full result with answer, evaluation, and root causes

get_config() -> dict

• Get current pipeline configuration

update_top_k(top_k: int)

• Update retrieval parameter

update_temperature(temperature: float)

• Update generation parameter

---

RootCauseOracle

Core diagnostic engine for root cause analysis.

Constructor

RootCauseOracle(
    query_history_file: str = "./query_history.json"
)

Methods

analyze(signals: dict) -> dict

• Analyze query execution signals
• Returns: Diagnosis with root causes and fixes

get_report(last_n: int = None) -> dict

• Get system health report
• last_n: Analyze last N queries (None = all)

validate_fix(fix_id: str) -> dict

• Validate if a fix actually worked
• Returns: Before/after comparison

get_public_output(result: dict) -> dict

• Get clean, user-facing output
• Returns: Simplified diagnosis

---

⚙️ Configuration Options

Pipeline Configuration

config = {
    # Chunking
    "chunk_size": 500,        # Characters per chunk
    "chunk_overlap": 50,      # Overlap between chunks
    
    # Retrieval
    "top_k": 3,               # Number of chunks to retrieve
    "embedding_model": "huggingface",  # or "openai"
    
    # Generation
    "temperature": 0.7,       # 0.0 (deterministic) to 1.0 (creative)
    "model_type": "openai",   # or "groq"
    "model_name": "gpt-4",    # Specific model
    
    # Diagnostics
    "enable_evaluation": True,
    "enable_tracing": True,
    "log_dir": "./logs"
}

pipeline = RAGPipeline(**config)

Environment Variables

# API Keys
OPENAI_API_KEY=your_openai_key
GROQ_API_KEY=your_groq_key

# Optional
RAG_LOG_DIR=./logs
RAG_VECTOR_STORE=./vector_store

---

🎓 Core Concepts

1. Corpus Coverage vs Retrieval Failure

The Critical Distinction

Scenario	Corpus Coverage	Retrieval Failure
Problem	Info doesn't exist in docs	Info exists but not retrieved
Cause	Missing documents	Wrong top_k, bad embeddings
Fix	Add documents (unfixable by system)	Increase top_k (fixable)
Example	"What's the CEO's salary?" (not in docs)	"What's the company mission?" (in docs, not retrieved)

How RAG-ORACLE Detects This:

# Checks if missing concepts exist in corpus
corpus_check = retriever.check_concepts_in_corpus(missing_concepts)

if concepts_missing_from_corpus:
    # Corpus Coverage - unfixable
    diagnosis = "Expand corpus. Missing: [concepts]"
    is_unfixable = True
else:
    # Retrieval Failure - fixable
    diagnosis = "Increase top_k from 3 to 5"
    is_unfixable = False

---

2. Correct Abstention

What is it? When the RAG system correctly says "I don't know" because the information isn't in the corpus.

Why it matters?

• ✅ Good: System abstains when info is missing (honest)
• ❌ Bad: System hallucinates an answer (dangerous)

How RAG-ORACLE Handles It:

# Detects abstention language
abstention_phrases = [
    "does not contain enough information",
    "cannot be answered from the provided context",
    "not found in the context"
]

if abstention_detected and info_missing_from_corpus:
    # This is GOOD behavior - don't flag as failure
    outcome = "SUCCESS_WITH_RISK"
    explanation = "System correctly abstained. Consider expanding corpus."

Example:

Q: "What is the CEO's favorite color?"
A: "The provided context does not contain information about the CEO's favorite color."

Diagnosis: ✅ Correct Abstention (not a failure!)

---

3. Fix Validation

The Problem: You applied a fix. Did it actually work?

RAG-ORACLE's Solution: Automated before/after comparison

# Before fix
before_queries = ["query1", "query2", "query3"]
# Apply fix: top_k = 3 → 5
# After fix
after_queries = ["query4", "query5", "query6"]

# Validate
validation = oracle.validate_fix_by_queries(before_queries, after_queries)

print(validation)
# {
#   "fix_applied": "Increase top_k from 3 to 5",
#   "failure_rate_change": "-20%",  # 20% reduction in failures
#   "retrieval_recall_change": "+0.15",  # Recall improved by 0.15
#   "verdict": "Fix effective"
# }

Validation Criteria:

• ✅ Effective: Failure rate drops >5% OR recall improves >0.05
• ❌ Ineffective: Failure rate increases OR recall drops
• ⚠️ No Change: Metrics stay within ±5%

---

4. Cost Awareness

The Hidden Problem: Your RAG system works, but wastes money.

How RAG-ORACLE Detects Waste:

# Scenario: Retrieved 10 chunks, only used 2
cost_analysis = {
    "total_tokens": 5000,
    "wasted_tokens": 4000,  # 8 unused chunks
    "wasted_cost_usd": 0.0008,
    "unused_chunks_ratio": 0.8
}

# Diagnosis
if unused_ratio >= 0.4:
    diagnosis = "Cost Inefficiency"
    fix = "Reduce top_k from 10 to 3"
    explanation = "You're retrieving 10 chunks but only using 2. Reduce top_k to save costs."

Cost Optimization Triggers:

1. Unused Chunks: Retrieved 10, used 2
2. High Cost Per Token: Paying too much for little value
3. Metadata Overfetch: Retrieved 5 sources, used 1

Real Impact:

• Before: 10 chunks × 500 tokens = 5000 tokens
• After: 3 chunks × 500 tokens = 1500 tokens
• Savings: 70% token reduction

---

🔍 Root Cause Types

1. User Query Issues

• Typos: "What is the captial of France?" → "capital"
• Ambiguous Grammar: "The fear of failure" (statement, not question)
• Over-Constrained: "List exactly 5 items" (too rigid)

Fix: Query preprocessing, normalization

---

2. Corpus Coverage

• Missing Information: Required data not in documents
• Unfixable: System can't fix this—user must add documents

Fix: Expand corpus with relevant documents

---

3. Retrieval Configuration

• Low Recall: Info exists but not retrieved
• Cause: top_k too low, chunks too small

Fix: Increase top_k, adjust chunk_size

---

4. Retrieval Noise

• High Recall, Low Relevance: Retrieved too much junk
• Cause: top_k too high, poor filtering

Fix: Reduce top_k, improve chunk filtering

---

5. Embedding Mismatch

• Domain-Specific Failure: Generic embeddings don't understand domain jargon
• Cause: Using general-purpose embeddings for specialized domain

Fix: Switch to domain-specific embeddings or fine-tune

---

6. Prompt Design

• Good Retrieval, Bad Answer: Retrieved right info, generated wrong answer
• Cause: Prompt doesn't align with task

Fix: Rewrite system prompt

---

7. Hallucination Risk

• Ungrounded Claims: Answer contains facts not in retrieved text
• Cause: High temperature, creative generation

Fix: Lower temperature, enforce extractive answering

---

8. Generation Control

• Unsupported Claims: LLM making things up
• Cause: Temperature too high

Fix: Lower temperature from 0.7 to 0.3

---

9. Cost Inefficiency

• Wasted Tokens: Retrieving more than needed
• Cause: top_k too high for simple queries

Fix: Reduce top_k, compress context

---

10. Systemic Drift

• Performance Degradation: System worked before, failing now
• Cause: Config changes, corpus updates, model changes

Fix: Review recent changes, rollback if needed

---

🆚 RAG-ORACLE vs Alternatives

Tool	What It Does	Integration	RAG-ORACLE Advantage
LangSmith	Tracing & monitoring (what happened)	Complex setup	✅ 3-line integration + tells you WHY
RAGAS	Evaluation metrics (scores)	Requires pipeline	✅ Works with any RAG + actionable fixes
Manual Debugging	Trial-and-error	N/A	✅ Instant diagnosis instead of hours

The Key Difference:

• LangSmith/RAGAS: "Your faithfulness score is 0.3" ❓
• RAG-ORACLE: "Your faithfulness is low because corpus is missing info. Add documents about X, Y, Z." ✅

Use Together: LangSmith for tracing + RAGAS for metrics + RAG-ORACLE for diagnosis = Complete RAG observability

---

📦 Package Structure

rag-oracle/
├── rag_pipeline/
│   ├── root_cause_oracle.py    # Core diagnostic engine
│   ├── pipeline.py              # Main RAG pipeline
│   ├── evaluator.py             # Evaluation metrics
│   ├── retriever.py             # Retrieval component
│   ├── generator.py             # Generation component
│   └── ...
├── examples/
│   └── example_usage.py         # Usage examples
├── tests/
│   └── test_oracle.py           # Unit tests
├── README.md                    # This file
└── setup.py                     # Package setup

---

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

---

📄 License

MIT License - see LICENSE for details.

---

🙏 Acknowledgments

Built with:

• LangChain for RAG components
• OpenAI/Groq for LLMs
• HuggingFace for embeddings

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: [e-mail]

---

🎯 Roadmap

• Support for more LLM providers (Anthropic, Cohere)
• Web UI for diagnostics dashboard
• Automated fix application
• Multi-language support
• Enterprise features (team collaboration, audit logs)

---

Made with MUHAMMAD HASSAN SHAHBAZ by developers who are tired of debugging RAG systems manually.

RAG-ORACLE: Because your RAG system deserves a doctor, not just a thermometer.