Skip to main content

ragcheck

Catch RAG regressions in pull requests. Run your pipeline on a small set of questions with known answers, and fail CI when the right answer, its supporting text, or the right document stops coming back.

Changes that break a RAG pipeline rarely look dangerous. Here someone trims every document to its first sentence "to save tokens". The code review is easy, the tests pass, and the assistant stops answering four questions out of five:

$ ragcheck-ci --dataset questions.jsonl --predictions predictions.jsonl --k 3
ragcheck: 5 cases, retrieval cutoff k=3

  Answer pass rate     20.0%  needs 100.0%  below
  Grounded pass rate   20.0%  needs 100.0%  below
  Retrieval recall@3   60.0%  needs 100.0%  below

Cases needing attention:
  refund-window
    missing answer phrase: "30 days"
    missing evidence phrase: "30 days"
  colombo-delivery
    missing answer phrase: "2 working days"
    missing evidence phrase: "2 working days"
    missing relevant docs: shipping-times
  cash-on-delivery
    missing answer phrase: "cash on delivery"
    missing evidence phrase: "cash on delivery"
    missing relevant docs: payment-methods
  delete-account
    missing answer phrase: "Settings"
    missing evidence phrase: "Settings"

RAG checks failed: 3 of 3 metrics below their threshold.

That's the pipeline in examples/ with RAGCHECK_DEGRADE=1. See it block a real pull request in ragcheck-demo.

The checks are deterministic: no LLM judge, no API key, and the same inputs always give the same result, so a failure in CI means something changed in your pipeline.

Install

pip install ragcheck-ci

No dependencies beyond Python 3.10+. (ragcheck was taken on PyPI, so the package and command are ragcheck-ci.)

1. Write down what good looks like

One JSON object per line, for questions your users really ask:

{"id": "refund-window", "question": "How long do customers have to request a refund?", "answer_contains": ["30 days"], "relevant_doc_ids": ["policy-refunds"]}
Field Meaning
answer_contains Phrases the answer must include.
evidence_contains Phrases the retrieved text must include. Defaults to answer_contains.
relevant_doc_ids Documents that should be in the top k results.

Ten to thirty questions covering your most important answers is enough to start.

2. Have your pipeline write what it did

For each question, your pipeline writes one line with the same id, its answer, and the chunks it retrieved, in order:

{"id": "refund-window", "answer": "You can request a refund up to 30 days after delivery.", "retrieved": [{"id": "policy-refunds", "text": "Refunds are handled by our support team. You can request a refund up to 30 days after delivery."}]}

examples/run_pipeline.py is a small working pipeline that does this. Replace its retrieval and answer steps with calls to your own.

3. Check it

ragcheck-ci --dataset questions.jsonl --predictions predictions.jsonl \
  --min-answer-pass-rate 0.95 --min-grounded-pass-rate 0.95 --min-retrieval-recall 0.9
Metric A case passes when
Answer pass rate every answer_contains phrase is in the answer
Grounded pass rate the answer passes and every evidence_contains phrase is in the retrieved text, so the answer is backed by what was retrieved
Retrieval recall@k the share of relevant_doc_ids found in the first k retrieved items

Exit code 0 = every metric meets its threshold, 1 = at least one is below, 2 = bad input (the message names the file, line or case). --json prints the full report.

In GitHub Actions

- uses: ShalomHunukumbura/ragcheck@v0
  with:
    dataset: eval/questions.jsonl
    predictions: /tmp/predictions.jsonl
    command: python eval/run_pipeline.py
    min-answer-pass-rate: "0.95"
    min-grounded-pass-rate: "0.95"
    min-retrieval-recall: "0.90"
    k: "5"

The Action runs your command with RAGCHECK_DATASET and RAGCHECK_PREDICTIONS set, then checks the result. Metrics below their threshold fail the check and show up as errors on the pull request, and the full table goes in the job summary.

Limits

  • Phrase matching is a regression signal, not a judge of correctness. A paraphrase ("one month" for "30 days") fails, and a long answer can contain the phrase and still say something wrong. Pick phrases that any correct answer must contain: numbers, names, steps.
  • It checks the questions you give it, nothing else.
  • If your pipeline isn't deterministic (sampling temperature above 0), lower the thresholds a little or results will flicker.

License

MIT

Metadata

Release files for ragcheck-ci 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ragcheck-ci 0.1.0
File Size Uploaded Provenance Actions
ragcheck_ci-0.1.0.tar.gz 21.6 kB With provenance Details for ragcheck_ci-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ragcheck-ci 0.1.0
File Interpreter ABI Platform Actions
ragcheck_ci-0.1.0-py3-none-any.whl Python 3 none any Details for ragcheck_ci-0.1.0-py3-none-any.whl

Total release size: 30.1 kB

Release files / ragcheck_ci-0.1.0.tar.gz

Download URL ragcheck_ci-0.1.0.tar.gz
Size 21.6 kB
Tags Source
SHA-256 checksum
How to use checksums
b1f7a21054220415edeb7200b74d63d2704b722978b510a4d05421d91f7e4648
BLAKE2b-256 checksum
How to use checksums
8e3a287133e145718953bcea404137a36379763cb72e5d3350bb833e26c77317
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Release files / ragcheck_ci-0.1.0-py3-none-any.whl

Download URL ragcheck_ci-0.1.0-py3-none-any.whl
Size 8.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
141388cd972d202d4aebc832dfacd74dca3d51deb09c51cf9115d6f60b0850f5
BLAKE2b-256 checksum
How to use checksums
9bfc174d4e03658c05d14fc82381784b5a0a4679bf16267c34d5f08ce2ae3439
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Security for ragcheck-ci 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ragcheck_ci-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page