RAGeval

Drop-in LLMOps Observability Platform for AI Applications

RAGeval provides self-hosted, research-grade evaluation and observability for Retrieval-Augmented Generation (RAG) and LLM pipelines. It introduces multi-judge consensus scoring, persona-aware evaluation, OpenTelemetry interoperability, and DSPy integration with minimal overhead.

Version: v0.1.10 | pip install omnismart-rageval

🔗 Live dashboard: https://rageval.ysiddo-ai-projects.app/ Self-hosting documentation: see SELF_HOSTING.md.

The 60-Second Setup

Evaluate interactions in real-time with a single decorator:

from rageval import track

@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def answer_question(query: str, context_chunks: list[str]) -> str:
    ...

Access the comprehensive analytics dashboard locally at http://localhost:8003.

Core Capabilities & Metrics

Competitive Differentiation

Quick Start

pip install omnismart-rageval     # CLI + import remain `rageval`
rageval init                      # creates ~/.rageval/rageval.db
rageval serve --port 8003

Integration Patterns

FastAPI

from rageval import track

@app.post("/ask")
@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def ask(query: str):
    chunks = await retriever.search(query)
    return await llm.generate(query, chunks=chunks)

LangChain

@track(model="groq/llama-3.3-70b-versatile")
def chain_invoke(query: str, context_chunks: list[str]):
    return chain.invoke({"query": query, "context": context_chunks})

Core API Endpoints

Quality & Reliability

Validated by a comprehensive test suite covering API functionality, evaluator consistency, decorator integration, and end-to-end scenarios.

pytest tests/ -q

Licensing & Commercial Use

RAGeval is open-source under the AGPL-3.0 License, ensuring it remains free for researchers, students, and open-source hobbyists.

Commercial Use: The AGPLv3 license mandates that any proprietary network service (e.g., SaaS, internal corporate tools) using or modifying this codebase must open-source its entire backend.

If you require integration into a closed-source commercial environment or need Enterprise features (e.g., SSO, VPC Deployment, Strict RBAC, PostgreSQL optimizations), you must obtain a Commercial License. See COMMERCIAL.md for details.

📡 Anonymous Telemetry

This project collects anonymous, GDPR-compliant startup pings to help understand usage volume and prioritize development.

• Data Collected: Startup event timestamp and anonymized deployment origin. No API keys, prompts, or sensitive data are collected.
• Opt-Out: Set TELEMETRY_OPT_OUT=true in your .env file.

See TELEMETRY.md for detailed privacy practices.