rag-eval-kit

Pluggable RAG evaluation framework. Run any RAG system against a labeled Q/A dataset and get accuracy, latency, and cost metrics across a configuration sweep.

Why rag-eval-kit?

Most teams end up writing one-off eval scripts per project: a loop that calls their RAG pipeline, compares outputs to a golden set, and prints a number. That works once — but the moment you want to sweep across top_k, temperatures, or chunking strategies, or compare LangChain vs LlamaIndex on the same dataset, those scripts turn into glue code. rag-eval-kit replaces that glue with a YAML config + a single RAGSystem protocol (one method: query), so the same eval loop drives any system, any metric, any config matrix — and spits out CSVs and charts teams can actually share.

Installation

pip install rag-eval-kit

# With optional dependencies:
pip install rag-eval-kit[anthropic]     # Anthropic Claude
pip install rag-eval-kit[openai]        # OpenAI
pip install rag-eval-kit[langchain]     # LangChain adapter
pip install rag-eval-kit[llamaindex]    # LlamaIndex adapter
pip install rag-eval-kit[r2r]           # SciPhi R2R server client
pip install rag-eval-kit[haystack]      # Haystack 2.x adapter
pip install rag-eval-kit[huggingface]   # HuggingFace datasets
pip install rag-eval-kit[cost]          # tiktoken for real pre-run cost estimates
pip install rag-eval-kit[all]           # Everything

For development:

git clone https://github.com/DennisKoshta/rag-eval-kit.git
cd rag-eval-kit
uv venv && uv pip install -e ".[dev,anthropic]"

Quick Start

1. Create a dataset (JSONL)

{"question": "What is the capital of France?", "expected_answer": "Paris"}
{"question": "Who wrote Romeo and Juliet?", "expected_answer": "William Shakespeare"}

JSONL and CSV files use fixed field names: question, expected_answer, and optionally expected_docs (list of strings) and tags (dict). For HuggingFace datasets, field names are configurable via question_field / answer_field / docs_field.

2. Write a config

dataset:
  source: jsonl
  path: ./data/questions.jsonl

system:
  adapter: raw
  adapter_config:
    llm_provider: anthropic
    llm_model: claude-sonnet-4-20250514

sweep:
  top_k: [3, 5, 10]
  temperature: [0.0, 0.3]

metrics:
  - exact_match
  - latency_p50
  - latency_p95
  - token_cost:
      pricing:
        claude-sonnet-4-20250514:
          input_per_1k: 0.003
          output_per_1k: 0.015

output:
  csv: ./results/run.csv
  charts: ./results/charts/
  html: ./results/report.html

3. Set API keys

rag-eval-kit reads API keys from environment variables or a .env file in the working directory:

export ANTHROPIC_API_KEY=sk-ant-...   # for Claude models
export OPENAI_API_KEY=sk-...          # for GPT / o-series models

Or copy .env.example to .env and fill in your keys. Shell-exported values take precedence over .env.

4. Run

rag-eval-kit run config.yaml

This expands the sweep matrix (3 top_k x 2 temperature = 6 configs), runs each against every question, and outputs:

• CSV with per-question scores and aggregate summary
• Charts (accuracy bars, latency box plots, cost vs accuracy scatter)
• Summary table printed to stdout

CLI Reference

rag-eval-kit run CONFIG        Run an evaluation sweep
  --dry-run                  Print plan without executing
  --output-dir DIR           Override output directory
  --filter TEXT              Filter configs (e.g. "top_k=5")
  --no-confirm               Skip cost confirmation prompt
  --verbose                  Show per-question results
  --seed N                   Inject reproducibility seed into adapter_config
  --concurrency N            Parallel queries per config (default 1)
  --checkpoint PATH          JSONL checkpoint for resumable runs

rag-eval-kit validate CONFIG   Validate a config file without running

rag-eval-kit report CSV_PATH   Re-generate charts from existing results
  --output-dir DIR           Output directory for charts
  --html PATH                Generate a self-contained HTML report

rag-eval-kit compare CSV_A CSV_B  Compare two results_summary.csv files
  --output PATH              Write comparison CSV
  --threshold FLOAT          Min absolute delta to flag (default 0.05)
  --html PATH                Write an HTML comparison report

Parallel sweeps

LLM calls are I/O-bound, so --concurrency 8 usually gets a ~6–8× speedup on real datasets with no code changes. Sweep configs still run sequentially (one at a time) so progress output stays readable; parallelism fans out across dataset items within each config. All bundled adapters are thread-safe — if you write a custom adapter, see docs/writing_adapters.md.

Resumable runs

Pass --checkpoint path.jsonl (or set output.checkpoint in YAML) and every completed (config, question) pair appends to that file immediately. If the process dies or you Ctrl-C out, re-running with the same checkpoint skips the completed items — no wasted tokens. The checkpoint records each row's config_params; if you edit your sweep between runs, mismatched rows are flagged and re-run.

HTML reports

Set output.html: ./results/report.html in your config (or pass --html to rag-eval-kit report) and get a single self-contained HTML file with sortable tables, inline charts, and a text filter for per-question results. No external CSS/JS — the file is fully portable and can be shared as-is.

Comparing runs

After tweaking a retriever and re-running, use rag-eval-kit compare to diff two result CSVs:

rag-eval-kit compare results_v1/results_summary.csv results_v2/results_summary.csv --html diff.html

Configs are matched by parameter equality. Each metric gets an absolute delta, percentage change, and directional indicator (improved/regressed/unchanged). Latency and cost metrics are direction-aware — a decrease is an improvement.

Tag-based breakdown

If your dataset includes tags on EvalItem (e.g. {"topic": "physics"}), rag-eval-kit automatically groups per-question scores by tag and reports per-tag averages in results_tags.csv and the HTML report's "Tag Breakdown" section. No config changes needed — tags are detected from the data.

Python API

from rag_eval_kit import RAGSystem, RAGResult, EvalDataset
from rag_eval_kit.config import load_config
from rag_eval_kit.orchestrator import run_sweep
from rag_eval_kit.reporters import write_csv, write_charts, write_html

config = load_config("config.yaml")
result = run_sweep(config, no_confirm=True)
write_csv(result, "results/")
write_charts(result, "results/charts/")
write_html(result, "results/report.html")

Custom RAG Systems

Implement the RAGSystem protocol -- a single query method:

from rag_eval_kit import RAGSystem, RAGResult

class MyRAGSystem:
    def query(self, question: str) -> RAGResult:
        # Your retrieval + generation logic here
        docs = my_retriever.search(question, top_k=5)
        answer = my_llm.generate(question, context=docs)
        return RAGResult(
            answer=answer,
            retrieved_docs=docs,
            metadata={
                "latency_ms": elapsed_ms,
                "prompt_tokens": usage.input_tokens,
                "completion_tokens": usage.output_tokens,
                "model": "my-model",
                "top_k": 5,
            },
        )

No inheritance required. Any object with a conforming query method works.

Metrics

Configuration Reference

Parametrized metrics

Most metrics are plain strings. Metrics that accept parameters use a single-key dict:

metrics:
  - exact_match                        # simple
  - precision_at_k:                    # with parameter
      k: 10
  - llm_judge:                         # LLM-based scorer
      provider: anthropic
      model: claude-sonnet-4-20250514
  - token_cost:
      pricing:
        claude-sonnet-4-20250514:
          input_per_1k: 0.003
          output_per_1k: 0.015

Retrieval metrics (precision_at_k, recall_at_k, hit_rate_at_k, mrr, ndcg_at_k) require dataset.docs_field to be set so ground-truth documents are available for comparison.

Adapters

Development

uv venv && uv pip install -e ".[dev]"
pytest                    # Run tests
ruff check .              # Lint
mypy rag_eval_kit/          # Type check

License

MIT © 2026 Dennis Koshta