RAGeval

Drop-in LLMOps observability. Self-hosted. SQLite-default. Persona-aware. Multi-judge consensus.

🔗 Live demo: https://rageval.ysiddo-ai-projects.app/demo/ · browser dashboard (score a query + view metrics). Also fully scriptable — API: /health, /eval/* via curl/HTTPie. On-demand backend (first request ~30–60 s to wake).

The 60-Second Pitch

from rageval import track

@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def answer_question(query: str, context_chunks: list[str]) -> str:
    ...

That's it. Open the dashboard at localhost:8003.

What It Measures

Comparison vs Alternatives

Quick Start

pip install omnismart-rageval     # distribution name; CLI + import remain `rageval`
rageval init                 # creates ~/.rageval/rageval.db
rageval serve --port 8003

Integration

FastAPI

from rageval import track

@app.post("/ask")
@track(model="anthropic/claude-sonnet-4-6", persona="cfo")
async def ask(query: str):
    chunks = await retriever.search(query)
    return await llm.generate(query, chunks=chunks)

LangChain

@track(model="groq/llama-3.3-70b-versatile")
def chain_invoke(query: str, context_chunks: list[str]):
    return chain.invoke({"query": query, "context": context_chunks})

Endpoints

License

MIT