raggate

A thin, CI-gated evaluation gate for RAG & LLM systems. Point it at a golden set, pick pass/warn/fail bands, and it fails your build when answer quality regresses. Runs with no API key (lexical heuristic scorers) and upgrades to LLM-as-judge scoring when you set OPENAI_API_KEY.

Fail your CI when RAG/LLM answer quality drops below a threshold. The part RAGAS and DeepEval leave out: a hard pass / warn / fail gate you drop straight into a pull request.

It is deliberately small. It does not replace RAGAS or DeepEval — it is the drop-in gate you wire into GitHub Actions in five minutes, and it composes with them (see How it compares).

Why

Most teams test their prompts by eyeballing a few answers and shipping. Then real users arrive and quality quietly regresses with no alarm. raggate treats RAG quality like any other production concern: measured against a versioned golden set, and gated in CI.

Quickstart

pip install raggate            # core (heuristic mode, no key needed)
# or, for LLM-as-judge scoring:
pip install "raggate[openai]"

raggate init          # scaffolds ./evals (golden set, gates.yaml, target.py)
raggate run           # scores the golden set, prints a banded report
raggate gate          # same, but exits non-zero when a KPI metric FAILs

raggate run scores a tiny built-in example immediately:

  raggate — 6 case(s) · judge: heuristic
  ──────────────────────────────────────────────────────────────
  METRIC                  SCORE   TARGET   BAND
  ──────────────────────────────────────────────────────────────
  faithfulness            1.000    0.900   PASS
  answer_relevancy        0.715    0.800   WARN
  citation_support        1.000    0.850   PASS
  context_coverage        1.000    0.800   PASS
  answer_correctness      0.627    0.850   WARN
  ──────────────────────────────────────────────────────────────
  heuristic mode — lexical proxies, informational only (never blocks).

Point it at your system

raggate calls one function you own. Edit evals/target.py:

from my_app.rag import answer   # your existing RAG entrypoint

def run(question: str) -> dict:
    result = answer(question)
    return {
        "answer": result.text,
        "contexts": result.retrieved_chunks,   # list[str] — for coverage & faithfulness
        "citations": result.citations,         # optional
    }

The metrics (and what they actually measure)

Two metric names are chosen to match what this kit computes, which is a cheaper proxy than the same-sounding RAGAS metric — so the name does not overpromise. The LLM-judge path is a pointwise rubric score; the heuristic path is a lexical proxy and is informational only (never gates).

Verify a metric name against its source before quoting a number — context_coverage and citation_support are intentionally not the RAGAS metrics.

Bands, not a single threshold

LLM-as-judge scores are non-deterministic (sampling, position bias, judge-model dependence). A lone >= 0.80 gate flaps on that noise and teams learn to ignore it. Bands encode intent — edit evals/gates.yaml:

judge:              # pin the judge — scores are only comparable against the same one
  model: gpt-4.1-mini
  runs: 1           # >1 averages several calls per rating to damp variance
  temperature: 0.0

metrics:
  faithfulness:     { kpi: true, fail: 0.75, warn: 0.85, target: 0.90 }
  context_coverage: { kpi: true, fail: 0.60, warn: 0.70, target: 0.80 }

FAIL blocks the merge (KPI metrics only). WARN reports but passes. Diagnostic (non-KPI) metrics inform without gating. Gating is only enforced in openai mode (a judge is configured); heuristic mode never blocks. Bands absorb variance — they don't remove it — so pin the judge and raise runs for high-variance metrics.

Defaults are starting points for general-purpose RAG, not domain-calibrated. faithfulness carries the highest bar because it is the hallucination guardrail. For regulated domains, use the shipped high-stakes profile — Stanford RegLab found commercial legal RAG tools hallucinate 17–33% of the time, so general-purpose faithfulness/citation floors are unsafe there (note: raggate's faithfulness measures grounding in retrieved context, not legal correctness — a distinct concern, but the direction holds):

raggate gate --gates evals/gates.high-stakes.yaml   # faithfulness/citation floors ≥ 0.90

GitHub Action

raggate ships as a composite Action — the whole gate in three lines:

# .github/workflows/rag-quality.yml
name: rag-quality
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: abhay23-AI/raggate@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}   # omit for heuristic mode

The score table is written to the GitHub Actions run summary automatically (via $GITHUB_STEP_SUMMARY), so you see pass/warn/fail on the PR checks page without opening logs. Inputs: dir (default evals), gates, openai-api-key, python-version, version.

Prefer to run it yourself? It's just the CLI:

      - run: pip install "raggate[openai]"
      - run: raggate gate
        env: { OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} }

Without the OPENAI_API_KEY secret the gate runs in heuristic mode (informational, never blocks). Add the secret to make it a hard quality gate.

How it compares

raggate is a gate, not a framework. Use it with the others, not instead of them.

Where they're stronger: RAGAS/DeepEval have far more metrics and DeepEval has a hosted dashboard; promptfoo does prompt comparison and red-teaming. If you want depth or dashboards, use those for scoring — raggate is the thin gate that turns any of them into a merge-blocking check.

Prior art

raggate is inspired by these; if you want deep metrics or dashboards, use them — raggate is the thin gate that composes with them.

• RAGAS (Apache-2.0) — the reference library for RAG-specific metrics. raggate's LLM metrics approximate its constructs with pointwise judges; its faithfulness/context-recall definitions are the standard.
• DeepEval (Apache-2.0) — the broadest metric set and pytest-native gating. If you want assertions inside your test suite, use it.
• promptfoo (MIT) — declarative eval + CI action with RAG assertions.
• TruLens, Arize Phoenix (Elastic-2.0, source-available), MLflow LLM eval — tracing/observability platforms.

Design notes

1. Retrieval quality (context_coverage) is measured separately from generation — most "the LLM is dumb" bugs are retrieval misses.
2. The golden set is versioned and small. Ten sharp cases you trust beat a thousand you don't; grow it every time a bug reaches production.
3. Gating is enforced only in openai mode; heuristic mode (no key) is a smoke test and never blocks. Note citation_support and context_coverage are deterministic overlap metrics with no LLM path — they still gate in openai mode.

Roadmap

• Reranker-lift metric (retrieval score before/after rerank)
• LLM/NLI path for citation_support (toward ALCE-style precision/recall)
• Adversarial / prompt-injection test pack
• Multi-run majority (not just mean) aggregation
• HTML report + PR comment bot

Contributing

Issues and PRs welcome — see CONTRIBUTING.md. Every metric must run in heuristic mode (no API key) and ship with a test.

Maintainer

Built and maintained by Abhay Trivedi — I build production-grade AI systems and help teams put real eval gates around production RAG/LLM pipelines. Open an issue for the tool, or reach me on LinkedIn if you want help wiring evals into your stack.

License

MIT © Abhay Trivedi. See LICENSE.