ragcheck
Catch RAG regressions in pull requests. Run your pipeline on a small set of questions with known answers, and fail CI when the right answer, its supporting text, or the right document stops coming back.
Changes that break a RAG pipeline rarely look dangerous. Here someone trims every document to its first sentence "to save tokens". The code review is easy, the tests pass, and the assistant stops answering four questions out of five:
$ ragcheck-ci --dataset questions.jsonl --predictions predictions.jsonl --k 3
ragcheck: 5 cases, retrieval cutoff k=3
Answer pass rate 20.0% needs 100.0% below
Grounded pass rate 20.0% needs 100.0% below
Retrieval recall@3 60.0% needs 100.0% below
Cases needing attention:
refund-window
missing answer phrase: "30 days"
missing evidence phrase: "30 days"
colombo-delivery
missing answer phrase: "2 working days"
missing evidence phrase: "2 working days"
missing relevant docs: shipping-times
cash-on-delivery
missing answer phrase: "cash on delivery"
missing evidence phrase: "cash on delivery"
missing relevant docs: payment-methods
delete-account
missing answer phrase: "Settings"
missing evidence phrase: "Settings"
RAG checks failed: 3 of 3 metrics below their threshold.
That's the pipeline in examples/ with RAGCHECK_DEGRADE=1. See it block a
real pull request in ragcheck-demo.
The checks are deterministic: no LLM judge, no API key, and the same inputs always give the same result, so a failure in CI means something changed in your pipeline.
Install
pip install ragcheck-ci
No dependencies beyond Python 3.10+. (ragcheck was taken on PyPI, so the package and
command are ragcheck-ci.)
1. Write down what good looks like
One JSON object per line, for questions your users really ask:
{"id": "refund-window", "question": "How long do customers have to request a refund?", "answer_contains": ["30 days"], "relevant_doc_ids": ["policy-refunds"]}
| Field | Meaning |
|---|---|
answer_contains |
Phrases the answer must include. |
evidence_contains |
Phrases the retrieved text must include. Defaults to answer_contains. |
relevant_doc_ids |
Documents that should be in the top k results. |
Ten to thirty questions covering your most important answers is enough to start.
2. Have your pipeline write what it did
For each question, your pipeline writes one line with the same id, its answer, and the
chunks it retrieved, in order:
{"id": "refund-window", "answer": "You can request a refund up to 30 days after delivery.", "retrieved": [{"id": "policy-refunds", "text": "Refunds are handled by our support team. You can request a refund up to 30 days after delivery."}]}
examples/run_pipeline.py is a small working pipeline that
does this. Replace its retrieval and answer steps with calls to your own.
3. Check it
ragcheck-ci --dataset questions.jsonl --predictions predictions.jsonl \
--min-answer-pass-rate 0.95 --min-grounded-pass-rate 0.95 --min-retrieval-recall 0.9
| Metric | A case passes when |
|---|---|
| Answer pass rate | every answer_contains phrase is in the answer |
| Grounded pass rate | the answer passes and every evidence_contains phrase is in the retrieved text, so the answer is backed by what was retrieved |
| Retrieval recall@k | the share of relevant_doc_ids found in the first k retrieved items |
Exit code 0 = every metric meets its threshold, 1 = at least one is below, 2 = bad
input (the message names the file, line or case). --json prints the full report.
In GitHub Actions
- uses: ShalomHunukumbura/ragcheck@v0
with:
dataset: eval/questions.jsonl
predictions: /tmp/predictions.jsonl
command: python eval/run_pipeline.py
min-answer-pass-rate: "0.95"
min-grounded-pass-rate: "0.95"
min-retrieval-recall: "0.90"
k: "5"
The Action runs your command with RAGCHECK_DATASET and RAGCHECK_PREDICTIONS set, then
checks the result. Metrics below their threshold fail the check and show up as errors on the
pull request, and the full table goes in the job summary.
Limits
- Phrase matching is a regression signal, not a judge of correctness. A paraphrase ("one month" for "30 days") fails, and a long answer can contain the phrase and still say something wrong. Pick phrases that any correct answer must contain: numbers, names, steps.
- It checks the questions you give it, nothing else.
- If your pipeline isn't deterministic (sampling temperature above 0), lower the thresholds a little or results will flicker.
License
MIT
Metadata
Release files for ragcheck-ci 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ragcheck_ci-0.1.0.tar.gz | 21.6 kB | With provenance | Details for ragcheck_ci-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ragcheck_ci-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for ragcheck_ci-0.1.0-py3-none-any.whl |
Total release size: 30.1 kB
Release files / ragcheck_ci-0.1.0.tar.gz
| Download URL | ragcheck_ci-0.1.0.tar.gz |
|---|---|
| Size | 21.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b1f7a21054220415edeb7200b74d63d2704b722978b510a4d05421d91f7e4648
|
|
BLAKE2b-256 checksum How to use checksums |
8e3a287133e145718953bcea404137a36379763cb72e5d3350bb833e26c77317
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logRelease files / ragcheck_ci-0.1.0-py3-none-any.whl
| Download URL | ragcheck_ci-0.1.0-py3-none-any.whl |
|---|---|
| Size | 8.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
141388cd972d202d4aebc832dfacd74dca3d51deb09c51cf9115d6f60b0850f5
|
|
BLAKE2b-256 checksum How to use checksums |
9bfc174d4e03658c05d14fc82381784b5a0a4679bf16267c34d5f08ce2ae3439
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logSecurity for ragcheck-ci 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for ragcheck_ci-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.