RAGeval

Drop-in LLMOps observability for RAG pipelines. Self-hosted. SQLite-default. Persona-aware. Multi-judge consensus.

Version: v0.1.27 | pip install omnismart-rageval

Live demo: https://rageval.ysiddo-ai-projects.app/ — browser dashboard (score a query + view metrics). Also fully scriptable — API: /health, /eval/* via curl/HTTPie. Self-hosting: see SELF_HOSTING.md.

The 60-Second Pitch

from rageval import track

@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def answer_question(query: str, context_chunks: list[str]) -> str:
    ...

That's it. Open the dashboard at localhost:8003.

What It Measures

Metric	Definition
Retrieval relevance	Cosine similarity between query and retrieved chunks (BGE-large by default)
Groundedness consensus	Multi-judge LLM scoring across your configured `JUDGE_MODELS` (min. 2 — no single-judge fallback), flags disagreement
Faithfulness	Per-sentence max-similarity to any chunk (NLI proxy)
Cost	USD per interaction, tracked by model
Latency	End-to-end wall-clock

See RESEARCH.md for the reasoning behind an LLM-judge, multi-judge-consensus design, and eval/JUDGE_BENCHMARK.md for measured accuracy against a hallucination benchmark.

Comparison vs Alternatives

Feature	RAGeval	Phoenix	Langfuse	TruLens
Self-hosted	✅	✅	✅	✅
SQLite default	✅	❌	❌	❌
Drop-in decorator	✅	partial	❌	partial
Persona-aware	✅	❌	❌	❌
Multi-judge consensus	✅	❌	❌	❌
Cost tracking	✅	✅	✅	partial
Setup time	60 sec	10 min	15 min	10 min

As of 2026, based on each project's public documentation. Feature sets change fast in this space — worth re-checking before you decide.

Quick Start

pip install omnismart-rageval   # v0.1.27 — distribution name; CLI + import remain `rageval`
rageval init                    # creates ~/.rageval/rageval.db
rageval serve --port 8003

Integration

FastAPI

from rageval import track

@app.post("/ask")
@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def ask(query: str):
    chunks = await retriever.search(query)
    return await llm.generate(query, chunks=chunks)

LangChain

@track(model="groq/openai/gpt-oss-120b")
def chain_invoke(query: str, context_chunks: list[str]):
    return chain.invoke({"query": query, "context": context_chunks})

Endpoints

Method	Path	Purpose
GET	/health	Liveness
POST	/eval/log	Score + store
POST	/eval/score	Score only (no storage)
GET	/eval/metrics?days=7	Aggregate dashboard data
GET	/eval/queries	Query log (filter by needs_review)
GET	/eval/cost-report?days=30	Cost breakdown by day + model
GET	/eval/alerts	Recent flagged queries
GET	/eval/events	Recent evaluation-pipeline telemetry
GET	/eval/config	Judge/embedding/threshold configuration
POST	/eval/retrieval-bench	A/B compare retrieval strategies
POST	/eval/embedding-comparison	Compare embedding models
WS	/eval/live	Real-time event feed

Full reference with request/response shapes: the in-app API Docs page (/api-docs in the dashboard).

Tests

70+ test functions across smoke, API, evaluator, decorator, DSPy integration, store, and e2e:

pytest tests/ -q

License & Commercial Use

This project is open-source under AGPL-3.0 — free for researchers, students, and open-source use.

The AGPLv3 requires that any proprietary network service (SaaS, internal corporate tooling) that uses or modifies this code also open-source its entire backend. If you need to use RAGeval in a closed-source commercial environment, or need enterprise features (SSO, custom RBAC, etc.), see COMMERCIAL.md for a commercial license.

Anonymous Telemetry

RAGeval sends a single anonymous startup ping, at most once per ~6 hours per running instance: a timestamp plus a randomly generated install ID (not derived from any hardware identifier) — no API keys, prompts, judge scores, or application data. Destination is the TELEMETRY_URL env var; set TELEMETRY_OPT_OUT=true in your .env to disable it outright (no request is made, not even a DNS lookup).

Metadata

Release files for omnismart-rageval 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for omnismart-rageval 0.2.1
File	Size	Uploaded
omnismart_rageval-0.2.1.tar.gz	54.0 kB	Sep 24, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for omnismart-rageval 0.2.1
File	Interpreter	ABI	Platform	Reset
omnismart_rageval-0.2.1-py3-none-any.whl 42.4 kB Sep 24, 2026	Python 3	none	any	Details

Total release size: 96.4 kB

Release files / omnismart_rageval-0.2.1.tar.gz

Download URL	omnismart_rageval-0.2.1.tar.gz
Size	54.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`a44469c2d0d5b3ba2d28117ac195738fc42273bd41af4b9a8cdce66d6f137cd4`
BLAKE2b-256 checksum How to use checksums	`b93980fd7b2393c2426af2cd7ad9e9cd6f82ed52baeb287911d6f593e46fb9d0`
Upload date	Sep 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.3`

Release files / omnismart_rageval-0.2.1-py3-none-any.whl

Download URL	omnismart_rageval-0.2.1-py3-none-any.whl
Size	42.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ac524c86078c4039b097180453d6cd829c884d8cfc429082071d3fb68e88b508`
BLAKE2b-256 checksum How to use checksums	`d6f9b21e994e68e0e372ac8ff30e0bd93c411ad4fefd7dae74f9259cec1f2da9`
Upload date	Sep 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.3`