rageval-ai

Drop-in RAG evaluation for Python — no server required. Evaluate any RAG/LLM output with one function call using LLM-as-judge methodology. Bring your own API key (OpenAI, OpenRouter, Azure, Anthropic), get faithfulness, hallucination, answer relevancy and 8+ more metrics back as a dict.

from rageval_sdk import evaluate

result = evaluate(
    question="What is the capital of France?",
    answer="Paris.",
    contexts=["Paris is the capital of France."],
)
print(result["overall_score"])         # 0.95
print(result["hallucination_score"])   # 0.02

That's it. No Docker, no SaaS account, no opaque internal LLM judge — your key, your model, your data.

---

Why rageval-ai?

	rageval-ai	Ragas	DeepEval	TruLens
Install & run in 3 lines	✅	⚠️	⚠️	⚠️
No server / SaaS required	✅	✅	✅	✅
Bring-your-own LLM (any OpenAI-compatible endpoint)	✅	⚠️	⚠️	⚠️
OpenRouter / Azure / Anthropic out of the box	✅	⚠️	⚠️	⚠️
Two-stage judge (cheap+strong models)	✅	❌	❌	❌
Background / non-blocking evaluation	✅	❌	⚠️	⚠️
Self-hosted FastAPI server option	✅	❌	❌	❌
Webhook callbacks	✅	❌	❌	❌
LangChain callback handler	✅	⚠️	⚠️	✅
Type hints (PEP 561)	✅	⚠️	⚠️	⚠️
Single dependency (httpx)	✅	❌	❌	❌

✅ = first-class, ⚠️ = possible but more setup, ❌ = not supported.

Use rageval-ai when you want a zero-ceremony way to score RAG outputs from a script, notebook, or production pipeline without adopting a new framework.

---

Installation

pip install rageval-ai

Requires Python 3.10+. The only runtime dependency is httpx.

---

Mode 1: Local Evaluation (No Server)

Set your API key once:

export OPENAI_API_KEY="sk-your-key"

Single Trace (3 lines)

from rageval_sdk import evaluate

result = evaluate("What is the capital of France?", "Paris.", ["Paris is the capital of France."])
print(result["overall_score"])  # 0.95

Batch Evaluation

from rageval_sdk import evaluate_batch

results = evaluate_batch([
    {"question": "What is RAG?", "answer": "Retrieval-Augmented Generation.", "contexts": ["RAG combines retrieval with generation."]},
    {"question": "What is Python?", "answer": "A programming language.", "contexts": ["Python was created by Guido van Rossum."]},
])

for r in results:
    print(f"Score: {r['overall_score']}")

Custom Provider (OpenRouter, Azure, etc.)

from rageval_sdk import evaluate, EvalConfig

config = EvalConfig(
    api_key="sk-or-...",
    base_url="https://openrouter.ai/api/v1",
    stage_1_model="qwen/qwen3-235b-a22b-2507",
    stage_2_model="qwen/qwen3-32b",
)

result = evaluate("Question?", "Answer.", ["Context."], config=config)

See examples/ for ready-to-run LangChain, LlamaIndex and OpenRouter notebooks.

---

Mode 2: Self-Hosted Server

Deploy the FastAPI evaluation server on your own infrastructure, then send traces from any client.

1. Deploy Server

git clone https://github.com/CYBki/llm-evaluation.git
cd llm-evaluation

# Configure
cp .env.example .env
nano .env  # set your OPENAI_API_KEY and other settings

# Start
docker compose up -d

# Verify
curl http://localhost:8000/health

2. Send Traces via SDK

from rageval_sdk import RagEvalClient

client = RagEvalClient(
    api_url="http://your-server:8000",
    api_key="your-api-key",
)

# Submit trace for evaluation
result = client.ingest(
    question="What is the capital of France?",
    answer="The capital of France is Paris.",
    contexts=["Paris is the capital and largest city of France."],
)

# Get evaluation results
trace = client.get_trace(result["id"])
print(trace["evaluation"]["overall_score"])

3. Auto-Evaluate in Your RAG Pipeline

from rageval_sdk import RagEvalClient

client = RagEvalClient(api_url="http://your-server:8000", api_key="key")

def handle_query(query):
    answer, contexts = my_rag_pipeline(query)  # your existing code

    # Non-blocking: sends to server for background evaluation
    client.ingest(question=query, answer=answer, contexts=contexts)

    return answer  # user gets answer immediately

4. Webhook Notifications

client.ingest(
    question="Q",
    answer="A",
    contexts=["C"],
    webhook_url="https://your-app.com/webhook",  # results POSTed here when ready
)

---

Background Evaluation (Local, Non-blocking)

from rageval_sdk import RagEvaluator

evaluator = RagEvaluator(api_key="sk-...", max_workers=4)

for query in user_queries:
    answer, contexts = my_rag_pipeline(query)
    evaluator.submit(question=query, answer=answer, contexts=contexts)

results = evaluator.wait()
evaluator.shutdown()

---

LangChain Integration

from langchain_openai import ChatOpenAI
from rageval_sdk import RagEvalCallback

callback = RagEvalCallback(api_key="sk-...")
llm = ChatOpenAI(callbacks=[callback])
# every LLM call in your chain is auto-evaluated in the background

A full RetrievalQA example is in examples/langchain_integration.py.

---

Evaluation Metrics

Metric	Description
overall_score	Weighted composite score (0-1)
hallucination_score	Detects fabricated information
faithfulness	Answer grounded in context
answer_relevancy	Answer relevance to question
context_precision	Quality of retrieved context
context_recall	Coverage of necessary information
clarity	Answer clarity
coherence	Answer coherence
helpfulness	Answer helpfulness
completeness	Answer completeness
citation_check	Source citation validation

---

API Reference

Local Mode

Function	Description
evaluate(question, answer, contexts)	Evaluate single trace (sync)
evaluate_batch(traces)	Evaluate multiple traces in parallel
evaluate_trace(question, answer, contexts, config=)	Async version
RagEvaluator(max_workers=4)	Background evaluator
EvalConfig(api_key=, base_url=, ...)	Custom configuration

Server Mode

Method	Description
RagEvalClient(api_url, api_key)	Connect to server
client.ingest(question, answer, contexts)	Submit trace
client.get_trace(trace_id)	Get results
client.list_traces(limit, offset)	List all traces
client.health()	Check server health

---

Roadmap

• v0.3 — Async batch API, retry/backoff, deterministic-mode for CI
• v0.4 — Native LlamaIndex evaluator
• v1.0 — Stable API, semantic-versioning guarantees

See CHANGELOG.md for release history.

---

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md.

---

License

MIT