ragtrace 0.2.0

Observability and tracing for Retrieval-Augmented Generation pipelines

RAGTrace 📊

Observability for RAG pipelines - Trace, inspect, and optimize Retrieval-Augmented Generation systems with ease.

✨ What is RAGTrace?

RAGTrace is a lightweight observability layer for RAG (Retrieval-Augmented Generation) systems that captures and visualizes every step of your pipeline:

• 🔍 Event Capture - Automatically intercepts retrieval, prompt, and generation events
• 💰 Cost Tracking - Accurate token counting and cost estimation per query
• 📊 Interactive Web UI - Modern timeline view with charts, filters, and event inspection
• 🔧 CLI Tool - Developer-friendly command-line interface
• 🌐 REST API - Query and analyze sessions programmatically
• 🧪 Regression Testing - Snapshot and compare RAG outputs

Think of it as OpenTelemetry, but specifically for RAG pipelines.

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/yourusername/ragtrace.git
cd ragtrace

# Install dependencies (using pip)
pip install -e .

# Or using Poetry
poetry install
poetry shell

# Initialize database
ragtrace init

Basic Usage

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from ragtrace import RagTracer

# Your existing RAG setup
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_texts(["Your documents here..."], embeddings)
llm = ChatOpenAI(model="gpt-3.5-turbo")

# Add RAGTrace - just one line!
tracer = RagTracer(auto_save=True)

# Create chain with tracer
chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    callbacks=[tracer]  # ← Automatic capture!
)

# Run your query (automatically captured)
result = chain.run("What is RAG?")

View Results

# View latest session in CLI
ragtrace show last

# List all sessions
ragtrace list

# Export to JSON
ragtrace export <session-id> > session.json

# Start API server (for Web UI)
ragtrace run

# Or start both servers for full Web UI experience
uvicorn api.main:app --port 8000 &  # API server
python ui/serve.py                   # UI server → http://localhost:3000

🌐 Web UI

RAGTrace includes a modern web interface for visualizing and analyzing your RAG pipelines:

Start the Servers

# Terminal 1: Start API server
cd ragtrace
source venv/bin/activate
uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

# Terminal 2: Start UI server
cd ragtrace
python ui/serve.py

Then open http://localhost:3000 in your browser.

Web UI Features

• 📋 Sessions View - Browse all captured RAG sessions with search and filtering
• 📊 Timeline View - Interactive timeline showing retrieval → prompt → generation flow
• 📈 Performance Charts - Waterfall chart for event durations, cost breakdown by component
• 🔍 Event Inspector - Click any event to see full details including tokens, costs, and data
• 🎯 Smart Filters - Filter events by type, duration, and cost
• 📤 Export Tools - Export session data as JSON or CSV, copy to clipboard

Quick UI Guide

1. Sessions Tab: View all RAG sessions, sorted by recent activity
2. Timeline Tab: Click on any session to see detailed event timeline
3. Click Events: Click timeline events to inspect full details
4. Apply Filters: Use dropdowns to filter by event type or performance metrics
5. Export Data: Use export buttons to download session data

📊 Example Output

╭─ Session: d4f3a8b2-... ─────────────────────────────────╮
│ Query: What is RAG?                                      │
│ Model: gpt-3.5-turbo                                     │
│ Cost: $0.00360                                           │
│ Duration: 1,850ms                                        │
╰──────────────────────────────────────────────────────────╯

┏━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Event       ┃ Duration   ┃ Cost       ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Retrieval   │ 150ms      │ $0.00001   │
│ Prompt      │ 0ms        │ $0.00000   │
│ Generation  │ 1,700ms    │ $0.00359   │
└─────────────┴────────────┴────────────┘

🎯 Features

✅ Core Features

• Automatic Event Capture - Works with LangChain callbacks
• Cost Tracking - Uses tiktoken for accurate token counting
• Timeline Visualization - See your RAG pipeline in action
• Session Management - Store and retrieve debugging sessions
• CLI Tool - Rich formatted terminal output
• REST API - FastAPI server with OpenAPI docs
• Web UI - Interactive timeline with charts and event inspection
• JSON/CSV Export - Export sessions for analysis
• Snapshot Testing - Save and compare pipeline outputs

🎨 Web UI Features

• Sessions Dashboard - View all sessions with search and sort
• Interactive Timeline - Click events to inspect details
• Performance Waterfall - Visualize event durations side-by-side
• Cost Breakdown Chart - See spending by component (embeddings, prompts, generation)
• Smart Filters - Filter by event type, duration, or cost
• Real-time Updates - WebSocket support for live session monitoring
• Export Tools - Download as JSON, CSV, or copy to clipboard

🎨 CLI Commands

ragtrace init              # Initialize database
ragtrace list              # List recent sessions
ragtrace show [id]         # View session details
ragtrace show last         # View latest  session
ragtrace export <id>       # Export to JSON
ragtrace clear             # Clear all data
ragtrace snapshot save     # Save snapshot
ragtrace snapshot list     # List snapshots
ragtrace run               # Start API server

🌐 API Endpoints

POST   /api/sessions                      # Create session
GET    /api/sessions                      # List sessions
GET    /api/sessions/{id}                 # Get session
PATCH  /api/sessions/{id}                 # Update session
DELETE /api/sessions/{id}                 # Delete session
POST   /api/sessions/{id}/events          # Add event
GET    /api/sessions/{id}/costs           # Get costs
POST   /api/snapshots                     # Create snapshot
GET    /api/snapshots                     # List snapshots
GET    /api/snapshots/{id1}/compare/{id2} # Compare snapshots

Visit http://localhost:8000/docs after running ragtrace run for interactive API documentation.

🏗️ Architecture

ragtrace/
├── core/              # Core business logic
│   ├── models.py      # Pydantic data models
│   ├── storage.py     # SQLite database layer
│   ├── cost.py        # Token counting & cost calculation
│   └── capture.py     # Event aggregation
├── langchain/         # LangChain integration
│   └── middleware.py  # Callback handler
├── api/               # REST API (FastAPI)
│   ├── main.py        # FastAPI application
│   └── routes.py      # API endpoints
├── ui/                # Web UI (HTML/CSS/JS)
│   ├── index.html     # Main UI page
│   ├── app.js         # Frontend application logic
│   ├── styles.css     # UI styling
│   └── serve.py       # Development server
├── cli/               # Command-line interface
│   └── main.py        # Click CLI commands
├── examples/          # Usage examples
│   └── simple_rag.py  # Complete working example
└── tests/             # Test suite
    ├── test_cost.py   # Cost calculation tests
    ├── test_storage.py # Database tests
    └── test_capture.py # Capture logic tests

📋 Requirements

• Python: 3.11+ (3.12 recommended)
• Dependencies: FastAPI, LangChain, tiktoken, Rich, Click
• OpenAI API Key: Required for examples (not for core functionality)

🛠️ Development

Setup Development Environment

# Clone repository
git clone https://github.com/yourusername/ragtrace.git
cd ragtrace

# Install with development dependencies
poetry install
poetry shell

# Run tests
pytest

# Run with coverage
pytest --cov=core --cov=langchain --cov=api --cov=cli

Running Tests

# All tests
pytest

# Specific test file
pytest tests/test_cost.py -v

# With coverage report
pytest --cov=core tests/

Code Quality

# Format code
black .

# Lint
ruff check .

# Type checking
mypy .

📖 Examples

Check out the examples/ directory for complete working examples:

• simple_rag.py - Basic RAG pipeline with debugger
• with_sources.py - RAG with source tracking (coming soon)

🔬 Use Cases

1. Debug Failed Queries

# See exactly why your RAG pipeline failed
ragtrace show last

2. Track Costs

# Monitor spending per query
ragtrace list --sort-by cost

3. Identify Retrieval Issues

# Check which documents were retrieved
session = tracer.get_latest_session()
print(session.retrieval_event.chunks)

4. Regression Testing

# Save baseline
ragtrace snapshot save "v1-baseline"

# Compare after changes
ragtrace snapshot compare <snapshot-id-1> <snapshot-id-2>

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with FastAPI
• Integrated with LangChain
• Token counting via tiktoken
• Beautiful CLI with Rich

📞 Support

• Issues: GitHub Issues
• Documentation: See examples/ directory
• API Docs: Run ragtrace run and visit http://localhost:8000/docs

🗺️ Roadmap

v0.2.0 (Current)

• Web UI for timeline visualization
• Interactive event inspection
• Performance charts (waterfall, cost breakdown)
• Export tools (JSON, CSV)
• Advanced regression testing
• LlamaIndex integration
• Prompt versioning

v0.3.0 (Planned)

• Agent tracing support
• Cost optimization suggestions
• Quality scoring
• Team collaboration features

v1.0.0

• Cloud mode
• Advanced analytics
• Alert system
• Multi-framework support

⭐ Star History

If you find RAGTrace useful, please consider giving it a star! ⭐

---

Built with ❤️ for RAG developers