ridi-check

See what a model or prompt update really changed in your assistant's answers, before you ship it.

A new model or an edited prompt can change what your assistant or agent answers without anyone noticing: average accuracy barely moves while individual answers flip. ridi-check runs the old and the new version on the same questions, more than once, and shows which answers changed and which of those changes hold on re-run. You need no test set, no answer key and no model judge.

What you will see · Who it is for · Start in one minute · Use it in CI · Connect your assistant · FAQ · Roadmap · Limitations · Reference

What you will see

In your terminal or CI log, ridi-check gate lists every answer that was right and is now wrong in every run, and exits 1 so the pipeline stops:

ridi-check gate: ridi-demo/ridi-out/results.csv, 30 cases, answer-key mode
stable_regression  case q02
stable_regression  case q09
stable_regression  case q18
stable_regression  case q25
counts: stable_regression=4 stable_improvement=1 regression_unconfirmed=0 improvement_unconfirmed=0 unstable=4 wrong_both=4 correct_both=17 incomplete=0
stable regressions: 4 (limit 0) - over the limit
gate: a limit was exceeded

In the report, each changed case shows the question, both answers, what the second run gave and the passages only one version used:

Every question ends in one group:

The report is one self-contained HTML file. It works offline, on a phone, in light and dark, in English and Arabic. The numbers above come from ridi-check demo, which uses synthetic data.

Who it is for

• You are about to swap the model, edit the prompt or change retrieval and want to know which answers move, not only whether an average moved.
• You have questions but no answer key. The report shows what changed; you judge only the changes that held on re-run.
• You want a CI check that fails when an answer that was right becomes wrong in every run.
• You only have answers in a spreadsheet. The browser version reads pasted cells and needs no install.

It is not for proving that answers are correct or that a system is ready to deploy, for agents that use tools or keep state between turns (not tested), or for explaining why an answer changed. See Limitations and the use cases.

Start in one minute

Python 3.10 or later, no dependencies. The demo needs no network and no API key; its data is synthetic.

pip install git+https://github.com/adeebnoor/ridi-check

ridi-check demo --open

Then check your own assistant:

ridi-check init my-check

It writes questions.csv, my_assistant.py (one function to fill in), ridi-check.json and ASK_YOUR_AI.md, a prompt that lets your AI coding assistant do the wiring. In that folder, try five questions with ridi-check run --limit 5, then run everything with ridi-check run. The user guide walks through each step and how to read the report.

No install at all: download web/ridi-update-check.html, open it and paste your results (CSV text or cells copied from a spreadsheet). The page makes no network request.

Use it in CI

- run: pip install git+https://github.com/adeebnoor/ridi-check
- run: ridi-check run --config ridi-check.json     # calls both versions, writes ridi-out/results.csv
- run: ridi-check gate ridi-out/results.csv        # exits 1 on a stable regression

A complete GitHub Actions workflow to copy: examples/ci/ridi-check.yml. gate prints counts and the offending cases; it never prints a verdict about your system. Limits such as --max-stable-regressions 2 are in the reference.

Connect your assistant

ridi-check has no model provider built in. Your assistant is the system under test, reached through one function:

def answer(question: str) -> dict:
    return {"answer": "text", "context": ["doc-12", "doc-40"]}   # context is optional

Ready-made examples in examples/adapters: an OpenAI-compatible HTTP API (no code), the Claude API through its official SDK, and a shell command with curl. Each reads its key from an environment variable and is tested against a local stand-in server.

The two versions should differ only in what the update changes. Details: developer guide.

FAQ

Do I need a test set or an answer key? No. Without an expected column the report shows which answers differ and you judge only the stable changes. With one, it speaks of regressions and improvements.

Does my data go anywhere? No. ridi-check calls only your assistant and writes local files. The report and the browser version make no network request.

Does it use a model to judge answers? No. A judge adds its own error. Scoring is exact match after normalisation, or a function you write.

How many questions and runs? 30 to 100 real user questions is a good first set. Two runs per version is the minimum for the stable label; use more when the decision is expensive. Calls = questions × 2 × runs.

Almost everything is "unstable" or "changed". Why? Your assistant words the same answer differently each time. Add expected answers with a scoring function, or use --similarity 0.9.

Which providers and frameworks does it support? Any. It talks to a function, a command or an HTTP endpoint, not to a provider.

Can I use it on an agent? For its question-in, answer-out behaviour, yes. Agents that use tools or keep state between turns have not been tested.

Roadmap

No dates and no promises; this is what is planned and what is not.

• Next: a release on PyPI and a hosted link for the browser version.
• Being considered: intervals for the rates, multi-turn conversations, tool-using agents, more report languages.
• Not planned: model-based judging, or any hosted service that receives your data.

Tell us what you need in an issue.

Limitations

• It finds change, not correctness. A version that is wrong the same way every time looks stable, and cases wrong in both versions are listed separately.
• Without an answer key or your judgment, a difference is a change, not a regression.
• Passages are shown as they differ and are not graded; an ungraded passage can still carry the evidence.
• Two runs is a thin estimate of re-run variation. More runs make the stable label firmer.
• It is not a readiness or safety certificate. No output says a system is fit to deploy; the gate prints counts and the decision stays yours.
• Not in this version: agents that use tools or keep state, model-based judging, statistical tests.

Documentation

• User guide: step by step, and how to read the report
• Use cases: twelve situations and how to set each one up
• Developer guide: adapters, files, CI, library use
• Reference: every command and option, results.csv, the labelling rules
• بالعربية: دليل الاستخدام · حالات الاستخدام · دليل المطورين

ridi-check follows the RIDI method.

Contributing

Bug reports, questions and pull requests are welcome: see CONTRIBUTING.md and the changelog. Report a vulnerability privately, as SECURITY.md describes.

Licence

MIT. See LICENSE.