RAGeval

Self-hosted LLMOps observability for RAG pipelines, with multi-judge consensus scoring, persona-scope detection, and a drop-in Python decorator.

pip install omnismart-rageval

Live application: rageval.ysiddo-ai-projects.app. Score queries and inspect consensus telemetry as it arrives.

• Research Background: RESEARCH.md / Online Research Documentation
• Empirical Benchmarks: BENCHMARK.md / Online Benchmark Dashboard
• User Guide: Online User Guide
• Architecture: ARCHITECTURE.md / Online Architecture
• Configuration reference: .env.example
• Self-Hosting Guide: SELF_HOSTING.md

Overview

from rageval import track

@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def answer_question(query: str, context_chunks: list[str]) -> str:
    ...

Instrumenting a RAG pipeline requires adding this decorator; the dashboard at localhost:8003 then reports groundedness, faithfulness, retrieval relevance, cost, and latency for every call.

What It Measures

The reasoning behind an LLM-judge, multi-judge-consensus design is in RESEARCH.md; measured accuracy against a hallucination-detection benchmark is in eval/JUDGE_BENCHMARK.md.

Comparison

Based on each project's public documentation as of 2026; feature sets in this space change quickly and are worth re-checking independently.

Quick Start

pip install omnismart-rageval
rageval init                    # creates ~/.rageval/rageval.db
rageval serve --port 8003

Integration

FastAPI

from rageval import track

@app.post("/ask")
@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def ask(query: str):
    chunks = await retriever.search(query)
    return await llm.generate(query, chunks=chunks)

LangChain

@track(model="groq/openai/gpt-oss-120b")
def chain_invoke(query: str, context_chunks: list[str]):
    return chain.invoke({"query": query, "context": context_chunks})

Endpoints

Full request/response schemas are documented on the dashboard's built-in API Docs page (/api-docs).

The API rejects request bodies larger than 1 MiB by default. Evaluation and comparison requests are also bounded to 10 queries, 20 chunks per query, 4,000 characters per text, and 200,000 total input characters. At most two expensive evaluation requests run at once per process by default; saturated requests receive HTTP 503 with Retry-After: 2. Tune RAGEVAL_MAX_REQUEST_BODY_BYTES and RAGEVAL_MAX_CONCURRENT_EVALUATIONS for the memory and CPU budget of the deployment. Provider calls use JUDGE_TIMEOUT (default 90 seconds, maximum 300) and remote embeddings use EMBED_TIMEOUT (default 30 seconds, maximum 120).

The API initializes and migrates its schema before serving requests. A failed schema bootstrap aborts startup rather than leaving a partially usable service; /health remains a liveness probe and /ready verifies that the embedding backend is warm and the database is available.

Tests

100+ test functions across smoke, API, evaluator, decorator, DSPy integration, storage, and end-to-end paths:

pytest tests/ -q

Support the project

If RAGeval is useful to you, a star on the repository helps other engineers find it, and following @Yacine-ai-tech is the simplest way to hear about new releases. Passing the link to a colleague who has the same problem is just as welcome.

Questions and field notes go to Discussions; reproducible bugs go to Issues.

License

Open-source under the AGPL-3.0 License, free for researchers, students, and open-source use. AGPLv3 requires that any proprietary network service built on modified RAGeval code also open-source its backend. A commercial license for closed-source use or enterprise features such as SSO and custom RBAC is available: see COMMERCIAL.md.

Anonymous Telemetry

On startup RAGeval sends one small ping, at most once every 30 seconds per instance, that contains the service name, the event name (startup), the version string and a randomly generated install identifier that is not derived from any hardware identifier. It never contains API keys, prompts, judge scores or application data. The receiving endpoint sees the request's network metadata, as any HTTP server does.

The ping is on by default and fires once when the API server starts or, for library use, on the first interaction logged in a process. TELEMETRY_URL redirects it to an endpoint you operate (an empty value means the default); TELEMETRY_OPT_OUT=true or DO_NOT_TRACK=1 disables it.