from ragarena import evaluate

evaluate(questions=[...], documents=[...],
         strategy="hybrid",                      # any of 18 strategies
         model="openai/gpt-4o-mini",             # swap with claude/gemini/llama/...
         embedding_model="voyage/voyage-3",      # swap with openai/cohere/jina/...
         metrics="quality").print_summary()

Why RagArena?

Choosing a RAG stack is guesswork today. "Is hybrid retrieval actually better than naive for my data? Is GPT-4o worth 17× the price of GPT-4o-mini for answer faithfulness? Do Voyage-3 embeddings beat OpenAI's on my legal corpus?"

RagArena turns those guesses into a leaderboard.

vs. RAG-Anything (LightRAG-based multimodal pipelines)

RAG-Anything is a great single-pipeline multimodal document processor (MinerU parsing, VLM captioning, multimodal knowledge graph). RagArena attacks a different — and complementary — problem: which RAG design should you ship?

Use both together: parse gnarly scanned PDFs with RAG-Anything/MinerU, feed the extracted content lists into RagArena's evaluate() to pick the best strategy/model combo before you commit.

Install

pip install ragarena                 # core
pip install "ragarena[all]"          # + all provider SDKs
export OPENAI_API_KEY=sk-...        # only the providers you use

60-second quickstart

from ragarena import evaluate

docs = [
    {"text": "Retrieval-Augmented Generation (RAG) grounds LLM answers in your documents."},
    {"text": "Hybrid retrieval combines dense vector search with BM25 keyword search."},
    {"text": "Cross-encoder rerankers like bge-reranker significantly improve precision."},
]

report = evaluate(
    questions=["What is RAG?", "How does hybrid retrieval work?"],
    documents=docs,
    reference_answers=["RAG grounds LLMs in docs", "It fuses dense + BM25 search"],
    strategy="hybrid",                          # naive|hybrid|multi_query|hyde|rerank|
                                                # rag_fusion|compression|crag|self_rag|
                                                # decomposition|step_back|agentic|flare
    model="openai/gpt-4o-mini",
    embedding_model="openai/text-embedding-3-small",
    metrics="quality",                          # quick|quality|full|production
)

report.print_summary()
report.save("report.json")

╭─ RagArena · hybrid · openai/gpt-4o-mini
├─ embedding : openai/text-embedding-3-small
├─ samples   : 2   wall time 6.4s
├──────────────────────────────────────────────────────────
│     faithfulness : 0.92      answer relevance : 0.95
│ context precision : 0.83      context recall : 0.88
│       hit rate : 1.0                   mrr : 0.75
│   avg latency : 3.1s           total cost : $0.00214
╰──────────────────────────────────────────────────────────

🏆 Find the best strategy/model in one call

from ragarena import compare

result = compare(
    questions=my_questions,
    documents=my_docs,
    reference_answers=ground_truth,
    configs=[
        {"strategy": "naive",       "model": "openai/gpt-4o-mini"},
        {"strategy": "hybrid",      "model": "openai/gpt-4o-mini"},
        {"strategy": "hyde",        "model": "openai/gpt-4o-mini"},
        {"strategy": "agentic",     "model": "groq/openai/gpt-oss-120b"},
        {"strategy": "hybrid",      "model": "anthropic/claude-3-haiku-20240307"},
    ],
)
result.print_leaderboard(sort_by="faithfulness")
print("WINNER:", result.best("faithfulness"))

The document index is embedded once and shared across all configs — comparisons are fast and cheap.

🎯 "Which strategy is best for MY data?"

Don't want to hand-pick configs? recommend_strategy() runs every strategy (or a chosen subset) against your corpus and questions, then ranks them by a quality/cost/latency-weighted composite score:

from ragarena import recommend_strategy

rec = recommend_strategy(
    questions=my_questions,
    documents=my_docs,
    reference_answers=ground_truth,
    model="groq/openai/gpt-oss-20b",
    embedding_model="google/gemini-embedding-001",
    quality_weight=0.7, cost_weight=0.15, latency_weight=0.15,   # tune for your priorities
)
rec.print_summary()
print(rec.best, rec.reasoning)

Also available as ragarena recommend --documents ... --questions ... (CLI) and POST /api/recommend (used by the Recommend tab in the playground UI).

📄 Bring your own data — any format, any database

from ragarena import parse_file, parse_dir, from_sql

docs = parse_file("report.pdf")                 # pdf, docx, pptx, html
docs += parse_file("notes.docx")                 # csv/tsv, json/jsonl, xml, yaml/yml,
docs += parse_file("sheet.xlsx")                 # xlsx (all sheets), md, txt, images,
docs += parse_dir("./knowledge_base/")           # .sql dumps, sqlite/sqlite3/db files — mixed dirs OK

docs += from_sql("postgresql://user:pass@host/db",       # any SQLAlchemy-supported DB
                  "SELECT id, title, body FROM articles")

Structured data is parsed retrieval-aware: table rows are chunked in groups that repeat the column headers ("col: value" pairs), JSON/YAML are flattened to full key-path facts (offices[0].city: Berlin), and XML keeps tag-path + attribute context — so embeddings stay self-contained and answers cite the right field. pip install "ragarena[ingest,sql]" for the optional parser/DB-driver dependencies (SQLite works with zero extras).

🖥 Web dashboard

RagArena serve            # → http://localhost:4000

• Overview — run history & framework stats
• New Evaluation — pick strategy/models from dropdowns, paste corpus, score instantly
• Compare — build config matrices, get leaderboards + Chart.js visualizations
• Runs — drill into every sample: answer, chunks, metric reasoning
• Catalog — browse all 100+ models with pricing/context windows

🔀 Every popular provider, one syntax

Models are addressed as provider/name:

from ragarena import completion

completion(model="openai/gpt-4o-mini", ...)          # OpenAI
completion(model="anthropic/claude-3-5-sonnet-20240620", ...)
completion(model="google/gemini-1.5-flash", ...)
completion(model="deepseek/deepseek-chat", ...)      # 97% cheaper than gpt-4o
completion(model="groq/openai/gpt-oss-20b", ...)   # sub-second inference
completion(model="ollama/llama3.1", ...)             # local & free
completion(model="bedrock/meta.llama3-1-405b-instruct-v1:0", ...)

Browse everything from the CLI:

RagArena models list                       # all 100+
RagArena models list --modality embedding  # embeddings only
RagArena models providers                  # provider summary
RagArena strategies                        # the 18 strategies

🧪 The 18 built-in strategies

🕸️ Graph RAG (dual-level retrieval)

graph_* strategies layer a lightweight knowledge graph over your index — entities are extracted per chunk, chunks that share entities form communities, and queries are answered at two levels:

• local (graph_local) — match the query's entities to graph nodes and pull the connected chunks. Best for precise "who/what" factual lookups.
• global (graph_global) — summarise each community, rank communities by relevance to the query, then synthesise a cross-document answer. Best for "how/why" analysis.
• hybrid / mix (graph_hybrid, graph_mix) — combine both levels.

The graph is built lazily and cached on a shared index, so compare() only builds it once. Entity extraction falls back to a deterministic keyword extractor if the LLM is unavailable.

from ragarena import compare

result = compare(
    questions=["Who builds Pinecone?", "How do retrieval systems relate?"],
    documents=my_docs,
    configs=[
        {"strategy": "graph_local",  "model": "openai/gpt-4o-mini"},
        {"strategy": "graph_global", "model": "openai/gpt-4o-mini"},
        {"strategy": "graph_hybrid", "model": "anthropic/claude-3-haiku-20240307"},
    ],
)

Build a graph index directly for inspection:

from ragarena import VectorIndex, GraphIndex

vi = VectorIndex(embedding_model="openai/text-embedding-3-small")
vi.add_documents(my_docs)
g = GraphIndex(vi).build("openai/gpt-4o-mini")   # cache on the index
local  = g.local_search("What is Pinecone?", k=5, llm_model="openai/gpt-4o-mini")
chunks, theme = g.global_search("How do vector DBs compare?", k=5, llm_model="openai/gpt-4o-mini")

🖼️ Multimodal RAG

Tables, images and equations are kept intact (not sentence-split) and tagged with a doc_type so retrieval and generation can treat them differently:

from ragarena import MultimodalDocument, evaluate

docs = [
    MultimodalDocument(content="| model | params |", doc_type="table"),
    MultimodalDocument(content="E = mc^2",           doc_type="equation"),
    {"text": "RAG grounds LLMs in retrieved context.", "metadata": {"doc_type": "text"}},
]
evaluate(questions=["..."], documents=docs, strategy="multimodal",
         model="openai/gpt-4o-mini")

📐 Metrics

Presets: quick · quality · full · production — or cherry-pick:

metrics=["context_precision", "context_recall", "hit_rate", "mrr",   # retrieval
         "faithfulness", "answer_relevance", "answer_correctness",   # generation (LLM-judge)
         "latency_s", "cost_usd", "total_tokens"]                    # operational

Any model can be the judge: judge_model="anthropic/claude-3-haiku-20240307".

📦 Use inside your existing pipeline

Already have a RAG system? Score it directly:

from ragarena import VectorIndex
from ragarena.engine import EvalSample, MetricContext
from ragarena.metrics import resolve_metrics

# ...run YOUR pipeline to get answer + chunks...
sample = EvalSample(question=q, reference_answer=gt,
                    generated_answer=your_answer,
                    retrieved_chunks=[{"text": c} for c in your_chunks],
                    context="...", usage={...}, latency_s=..., intermediate={})
for m in resolve_metrics("full"):
    print(m.name, m.compute(sample, MetricContext(judge_model="openai/gpt-4o-mini")).score)

HTTP API

curl -X POST localhost:4000/api/evaluate -H 'Content-Type: application/json' -d '{
  "questions": ["What is RAG?"],
  "documents": [{"text": "RAG grounds LLM answers in documents."}],
  "strategy": "hybrid",
  "model": "openai/gpt-4o-mini",
  "embedding_model": "voyage/voyage-3"
}'

Also available: POST /api/compare · GET /api/catalog · GET /api/runs/{id} · GET /health

Roadmap

• Async batch runner + checkpoint/resume
• Statistical significance tests (paired bootstrap) between configs
• Chroma/Pinecone/Qdrant backends for VectorIndex
• Graph RAG incremental updates + community-aware re-indexing
• Cloud/blob storage ingestion (S3, Azure Blob, GCS) alongside local parse_dir()
• Prompt-optimization loop (DSPy-style)
• Team features: API keys, budgets, RBAC
• CI mode: ragarena ci --threshold faithfulness>=0.8 (fails PRs on regressions)

Contributing

PRs welcome! pip install -e ".[dev]" && pytest. Please read CONTRIBUTING.md.

License

MIT © RagArena contributors