ragwatch

ragwatch is a lightweight, zero-friction Python SDK for evaluating, monitoring, and detecting regressions in RAG (Retrieval-Augmented Generation) pipelines.

Think of it as a mini LangSmith or Braintrust — built from scratch.

Features

Installation

pip install ragwatch

Or install from source (editable mode):

git clone https://github.com/mohammedsohel2052/ragwatch
pip install -e ./ragwatch

Quick Start

1. Zero-Friction Monitoring with @monitor

from ragwatch import monitor

@monitor(project_name="my-rag-bot")
def generate_answer(query, docs):
    # Your existing RAG generation code — unchanged
    return {"answer": "...", "latency_ms": 120}

Every call now logs a trace to ragwatch.db and prints:

[RAGWatch] ✓ Traced 'generate_answer' | 452ms | project='my-rag-bot'

2. Full Evaluation Pipeline

from ragwatch import Evaluator

evaluator = Evaluator(db_path="eval_results.db", embed_fn=my_embed_fn)

summary = evaluator.evaluate(
    golden_dataset=dataset,           # list of {"question": ..., "expected_answer": ..., ...}
    retrieval_fn=my_retrieval_fn,     # fn(query) -> list[dict]
    generation_fn=my_generation_fn,   # fn(query, docs) -> {"answer": str, "latency_ms": float}
)

3. Regression Detection

# Check current run against the stored baseline
alerts = evaluator.check_regressions(summary)

# If happy with results, promote as the new baseline
evaluator.promote_to_baseline(summary["run_id"])

4. RAGWatch UI Dashboard (Mini-LangSmith)

Visualize your evaluations and live telemetry instantly. From your project root, run:

python -m ragwatch.ui

Then open http://localhost:5050 in your browser.

• Evaluations Tab: Track Precision, Recall, Relevancy, and Faithfulness over time.
• Live Traces Tab: See exactly what your @monitor decorator is capturing in real-time.

Metrics Explained

Project Structure

ragwatch/
├── ragwatch/
│   ├── __init__.py       # Public API
│   ├── evaluator.py      # Main Evaluator class
│   ├── scorer.py         # Metric calculation functions
│   ├── storage.py        # SQLite persistence (runs + traces)
│   ├── monitor.py        # @monitor decorator
│   └── regression.py     # Regression alert logic
└── pyproject.toml

Built By

Mohammed Sohel Patwari — GitHub

Inspired by RAGAS, LangSmith, and Braintrust.