rag-eval-harness

RAGAS with a memory and a diff view.

What this proved: a weaker retriever on a 40-question Nimbus help-center set cut mean context recall from 0.880 to 0.740. rag-eval regress --threshold 0.05 exited 1. Full write-up: Nimbus case study. 90-second recording: ./scripts/demo.sh.

Chunking the same corpus doubled lexical precision and failed the recall gate. Production stayed on whole articles. Chunking case study. ./scripts/demo-chunking.sh.

Upload a golden RAG dataset, run a pipeline (precomputed traces or HTTP), score with RAGAS, lexical overlap, or a keyless stub evaluator, persist runs, and fail CI when mean metrics drop versus a tagged baseline.

This is not a RAG framework. There is no ingestion or retrieval stack. Evaluation orchestration is the product.

10-minute path

Requires Python 3.11+ (uv recommended) and, for the UI, Node 20+.

git clone https://github.com/JohnCarl-30/rag-eval-harness.git && cd rag-eval-harness
uv sync
./scripts/demo.sh

You should see context_recall FAIL (0.880 → 0.740) and the script exit 0. That red rag-eval regress is the product. No API key. Then score dummy traces if you want a green run:

uv run rag-eval eval examples/dummy-rag/traces.jsonl --evaluator stub -o /tmp/rag-eval-head.json

Optional. Call a live HTTP SUT (the canned FAQ server):

python3 examples/dummy-rag/server.py --port 8080 &
uv run rag-eval eval examples/dummy-rag/golden.csv \
  --sut-url http://127.0.0.1:8080/query \
  --evaluator stub

UI (dev):

uv run rag-eval serve          # API on 127.0.0.1:8000, open on loopback (no API key)
cd web && npm install && npm run dev   # Vite on :5173, proxies /api

Open http://127.0.0.1:5173, upload examples/dummy-rag/traces.jsonl or golden.csv, create a run, tag a baseline, open Diff.

Docker:

cp .env.example .env          # set RAG_EVAL_API_KEY (required: the API binds 0.0.0.0)
docker compose up --build
# UI+API: http://127.0.0.1:8000   dummy RAG: http://127.0.0.1:8080/query

Install

pip install git+https://github.com/JohnCarl-30/rag-eval-harness.git
pip install 'rag-eval-harness[ragas]'   # after a PyPI release, or: pip install '.[ragas]' from a clone

rag-eval eval traces.jsonl --evaluator stub
rag-eval eval golden.csv --sut-url https://your-rag.example/query --sut-token "$TOKEN"
rag-eval baseline <run-id>
rag-eval regress --baseline <run-id-or.json> --head <run-id-or.json> --threshold 0.05
rag-eval diff --baseline <run-id-or.json> --head <run-id-or.json>
rag-eval investigate --baseline <run-id-or.json> --head <run-id-or.json>   # needs [agent]
rag-eval investigate ... --mode supervisor --trace trace.json
rag-eval serve

regress exits 1 if any mean drops by more than --threshold (default 0.05), or if head has more errored rows than baseline (means skip errored rows, so a timing-out SUT would otherwise pass). diff prints the same means plus the worst per-row drops (rows paired by question) and always exits 0 on a successful compare.

investigate runs the regress gate, then a LangGraph fan-out of Pydantic AI agents explains a failure: one diagnosis per failing stage (retrieval, generation), then a summary. --mode supervisor swaps the fixed graph for one agent that picks its own tools. --trace writes every agent run as JSON; --otel sends spans via Logfire. Same exit codes as regress. Install with pip install 'rag-eval-harness[agent]'. See docs/agent.md.

Contracts

Docs: adapters · metrics · CI · investigator · storage · Nimbus case study · Chunking case study · Cost and p95 · Lexical vs judge

GitHub Action

stub is keyless CI smoke. Hashes look healthy on junk contexts. Use lexical for a real gate.

- uses: JohnCarl-30/rag-eval-harness@v0.2.0
  with:
    traces: tests/golden/traces.jsonl
    evaluator: lexical
    baseline: tests/golden/baseline.json
    threshold: "0.05"

Public consumer: Relaydesk fails PRs when Nimbus recall drops. See docs/ci.md.

Development

uv sync --extra agent   # without the extra, tests/test_agent.py is skipped
uv run pytest
uv run ruff check src tests
cd web && npm install && npm run build
docker compose up --build   # then ./scripts/smoke.sh

License

MIT. See CHANGELOG and CONTRIBUTING.