opik-rigor

Statistical assertions and pinned-judge evaluation primitives for LLM test suites.

assert_pass_rate(result, min_rate=0.9)   # not: assert pass_rate >= 0.9

Two primitives, done properly, with an audit trail. Optional Opik integration.

The problem

You have an eval. It calls a model 20 times, 18 pass, and your test asserts pass_rate >= 0.9. It goes green and you ship.

That test told you almost nothing, for three separate reasons.

You measured a stochastic system once. 18/20 is a sample, not a property. The same system on the same inputs gives you 17/20 tomorrow and the suite goes red with nothing having changed. So the team adds a retry, or drops the bar to 0.85, and now the gate is measuring the team's patience rather than the model.

Your judge moved. The model id was claude-3-5-sonnet-latest. The provider re-pointed the alias in March. Every score you recorded before March is not comparable to every score after, and nothing anywhere says so. Or somebody improved the wording of the rubric, which is the same problem wearing different clothes.

Your failures and your outages are in the same bucket. The provider 500ed four times, your harness counted those as failures, and now a quality gate is reporting an infrastructure incident. Nobody notices, because the number moved in a direction that looks like a real regression.

rigor fixes exactly these three things and nothing else.

The primitives

1. Statistical gates

An assertion that accounts for having sampled a stochastic system n times rather than measured it once. assert_pass_rate compares the one-sided Wilson lower confidence bound against your bar, never the observed rate.

The practical consequence is worth internalising before you use it:

You cannot pass a 90% gate by scoring 90%. The bound approaches the observed rate from below and never reaches it, so you need real headroom above the bar — and how much headroom is exactly what n buys you. That is the whole idea. A gate that let 18/20 through would be telling you a story about 20 coin flips.

Three gates ship:

• assert_pass_rate(result, min_rate=...) — Wilson lower bound vs a floor.
• assert_score_distribution(result, min_mean=..., min_p10=..., max_stddev=...) — each threshold independent and optional, every violation reported at once. A mean gate alone passes a system that is excellent four times in five and unusable the fifth time, which is the failure users actually notice.
• assert_no_regression(current, baseline) — Mann-Whitney U against a recorded baseline. Nonparametric because judge scores are ordinal and routinely multi-modal; a t-test there is testing an assumption the data does not meet.

The failure message is the statistical report. It distinguishes the two failures that matter, in as many words: you missed the bar versus you did not sample enough to tell.

Every gate takes a SampleResult, and there are two ways to get one:

• sample(fn, n) calls fn n times and records each run's value, outcome, duration and exception.
• sample_of(values) wraps results you already have in the same object — for gating a run you collected elsewhere, or feeding a stored baseline into the regression gate, without pretending to re-run either.

sample_of is not sample_over. sample_over(items, fn) — map a function across a dataset — is a roadmap item below and does not exist yet; sample_of takes finished values and no function at all. Two names one letter apart, meaning opposite things, is exactly the kind of thing to read twice.

2. A pinned judge

PinnedJudge refuses to run against an aliased model id — at construction, not after a week of wasted compute:

judge 'summariser' refuses unpinned model id 'claude-3-5-sonnet-latest'. It contains
the alias token 'latest', which names whatever the provider is serving today rather
than one fixed version. A pinned id names one immutable model version ... An alias
re-points over time, which silently invalidates every score recorded against it.

An id is pinned when it carries no alias token (latest, newest, current, stable, default) and ends in a release designator — a release number (claude-opus-5, claude-opus-4-8), a date stamp (claude-haiku-4-5-20251001, gpt-4o-2024-08-06), or an explicit version (-v1, -2.1.0). The property being checked is immutability, not spelling: a retired id that still names one fixed set of weights is pinned, and an id ending in a word (gpt-4o, mistral-large) is not, because a word names a kind of model and kinds get re-pointed. What no string can tell you is a provider's policy, so the one place that needs vendor knowledge — providers that publish <family>-<number> as a moving pointer — is a single documented table in pinning.py, and its limits are written down there.

It hashes its rubric and raises when the rubric changes underneath a baseline (accept_rubric_change=True acknowledges it and records both hashes). And it parses the judge's response strictly: an unparseable response raises, and is never converted into a failing verdict — missing data is not evidence of failure, and folding it into the failure bucket biases your pass rate by exactly the judge's own flakiness rate.

3. An evidence log you cannot edit

Everything above writes to an append-only JSONL log with a fixed envelope and no delete, truncate, or rotate method. That absence is a feature, and a test enforces it — an audit trail you can quietly edit is not an audit trail.

Quickstart

Every code block below was executed in a clean virtualenv holding nothing but the published opik-rigor 0.1.1 wheel from PyPI — the release this one supersedes — and every message, number and hash below is that run's own output. Every block that prints something was then re-executed unchanged against the 0.2.0 source tree on 2026-08-14 and reproduced byte for byte. The install measurements below were not re-taken, and name the release they were taken on. The only editing is hard-wrapping long lines to the page width. Nothing is elided, abbreviated, or retyped from memory — where an earlier revision of this file showed ... inside a hash, that was illustrative text under a claim of verbatim, and it is gone.

pip install opik-rigor

What that pulls in. opik-rigor's own code is 0.33 MB in 0.1.1, and 0.2.0 adds about 32 KB of source to that with the worked-example module described below. It requires NumPy and SciPy, which come to 180.6 MB on disk between them (numpy 2.5.2 and scipy 1.18.0 on CPython 3.14/Windows, counting the numpy.libs and scipy.libs directories that carry the bundled BLAS/LAPACK). NumPy is used by every gate. SciPy is used by exactly one function — assert_no_regression, for scipy.stats.mannwhitneyu — and is imported on first call rather than at package import, so a suite that never calls that gate never loads SciPy and does not pay for it at import time. Both stay required, so pip install opik-rigor continues to give you a working assert_no_regression.

What that costs you at the prompt, which is the number you feel: measured on 0.1.1 with pip install --no-cache-dir opik-rigor into an empty virtualenv (Python 3.14.4, Windows), 54 seconds, and the virtualenv grows from 11.5 MiB to 192.8 MiB across four packages. That is the disk figure above plus the interpreter's own baseline; nothing else is hiding in it. The [opik] extra costs a great deal more — see Optional extras before you add it.

A worked example rubric ships inside the package, so the install gives you something to point the judge at:

python -c "import opik_rigor, shutil; shutil.copy(opik_rigor.example_rubric_path(), 'rubric.md')"

Read it, then edit it into your own — a rubric is the measuring instrument, and one copied from a library measures the library's idea of quality rather than yours. It deliberately says nothing about JSON: PinnedJudge appends the response-format instruction to the prompt itself, so a rubric that restates it sends the same block twice. Then:

from opik_rigor import EvidenceLog, FakeAdapter, PinnedJudge, assert_pass_rate, sample

log = EvidenceLog("evidence.jsonl")
adapter = FakeAdapter(  # a real judge would be AnthropicAdapter("claude-...-20250929")
    responses=['{"pass": true, "score": 5}'] * 9 + ['{"pass": false, "score": 2}'],
    seed=1,
)
judge = PinnedJudge(adapter, "rubric.md", log, name="summariser")

result = sample(lambda: judge.evaluate("Summarise this.", "A summary."), 20, evidence=log)
assert_pass_rate(result, min_rate=0.9, evidence=log)

This fails, and the failure is the point:

opik_rigor.distribution.PassRateError: pass rate gate failed: 18/20 passed (observed
0.9000); one-sided 95% Wilson lower bound 0.7383 < min_rate 0.9000. Two-sided 95%
interval [0.6990, 0.9721]. The observed rate 0.9000 clears min_rate 0.9000 but the
lower bound does not: this is an underpowered sample, not a demonstrated failure.
20 runs cannot distinguish a system at 90.0% from one at 73.8%. The observed rate
sits exactly on min_rate, and the lower bound approaches the observed rate from
below without ever reaching it: no sample size clears this bar at exactly 0.9000.
The system needs real headroom above min_rate, or min_rate has to come down.

assert pass_rate >= 0.9 would have gone green on that sample.

Same judge, same seed, sampled properly and gated at a bar it can actually defend. Change 20 to 200 and min_rate to 0.8 — and keep the report the assertion hands back, because on success it prints nothing at all:

report = assert_pass_rate(result, min_rate=0.8, evidence=log)
print(
    f"passed={report['passed']}  observed={report['pass_rate']:.4f}  "
    f"lower_bound={report['lower_bound']:.4f}  min_rate={report['min_rate']}"
)

passed=True  observed=0.9150  lower_bound=0.8768  min_rate=0.8

Note report['pass_rate'] under the word observed. That mismatch is not a typo here — it is the roadmap's "the key names are not guessable" complaint in its sharpest form, and it is why this block now shows the print rather than a line of output with no way to produce it. The full key set on success is gate, label, passed, n, successes, failures, pass_rate, lower_bound, interval_lower, interval_upper, min_rate, confidence, method.

And the other primitive. Append the line Be stricter about caveats. to rubric.md — the exact text matters, because the second hash below is a hash of the result — then build the same judge against the same evidence log:

judge = PinnedJudge(adapter, "rubric.md", log, name="summariser")

opik_rigor.errors.RubricDriftError: rubric drift for judge 'summariser': evidence log
last recorded 556c1383350d73d71235e40c719cdf816bec8a5693cc4750ad01ae421128dc5d, rubric
file now hashes to 833331e342e2a6c05c4759621fde7b606fbb4a22dfadf283c35ca9d5176138c8.
Scores before and after this change are not comparable. Pass
accept_rubric_change=True to acknowledge and record the change.

The first hash is the rubric exactly as it ships in the wheel; the second is that file with the line above appended. Both are printed in full, because half a hash is not something you can compare against a log — the message never abbreviates one, and an earlier revision of this README showed ... in the middle of both.

A full worked example — corpus, judge, both gates, a baseline, a simulated regression, and the audit trail — ships inside the wheel and runs offline with no credentials:

python -m opik_rigor.examples.summarise_eval --seed 7 --n 40

A module, not a path, deliberately. This line used to read python examples/summarise_eval.py, and examples/ is a directory in the git tree that is not in the artifact — an install contains opik_rigor/ and opik_rigor-0.2.0.dist-info/ and nothing else — so the command this quickstart ended on could not be run by anyone who had followed the install instruction four paragraphs above it. The walkthrough of what it prints, screen by screen, is in the repository.

This one command is the exception to the paragraph at the top of this section. Everything else here was first run against the published 0.1.1 wheel, and opik_rigor.examples is not in 0.1.1 — 0.2.0 is the release that ships it, so this line was run against the 0.2.0 source tree rather than against a published wheel. If it reports No module named, the install predates 0.2.0 and pip install --upgrade opik-rigor is the fix rather than a checkout. A project's long description is also frozen at upload, so the page PyPI renders for an earlier version is that version's README — for 0.1.1, ellipses and broken paths and all — and no correction made here ever reaches it. This file self-heals in the repository; the index does not.

Optional extras

pip install "opik-rigor[opik]"     # log samples and verdicts to Opik
pip install "opik-rigor[pytest]"   # @pytest.mark.rigor_repeat, rigor_judge fixture

Opik — two functions, not a framework. They live in opik_rigor.integrations.opik, not at the package root:

from opik_rigor.integrations.opik import log_assertion_to_opik, log_sample_to_opik

Neither name is on the package root, and importing either from there raises ImportError. That is a deliberate arrangement rather than an oversight to be tidied away: opik_rigor's __init__ imports no integration at module scope — a test asserts it in a subprocess — so that import opik_rigor can never drag in a vendor SDK. Putting these two names on the package root is the one change that would break the property the last paragraph of this section is about.

log_sample_to_opik maps a sample to a trace with one span per run (a run that raised is visibly distinct from one that failed), and log_assertion_to_opik maps a gate's verdict to feedback scores. The verified API surface, the version bounds, the reasoning behind them, and a correction to a claim this project got wrong about Opik's own documentation are all in COMPATIBILITY.md.

"Two functions, not a framework" describes rigor's side of the seam, not the install. Measured on 0.1.1 the same way as the core install above, into its own empty virtualenv: pip install --no-cache-dir "opik-rigor[opik]" takes 4 minutes 48 seconds and produces a 414.9 MiB virtualenv holding 74 packages, against 54 seconds, 192.8 MiB and 4 packages for the bare install. What arrives with it includes litellm (101.9 MiB), openai, tokenizers, huggingface-hub, hf-xet, tiktoken, sentry-sdk, three tree-sitter grammars, pydantic, boto3 type stubs — and pytest, which Opik pulls in for its own plugin. Add the extra when you want the dashboard; do not add it because the sentence above made it sound small.

pytest — @pytest.mark.rigor_repeat(n=50, min_rate=0.9) runs a test n times and applies the gate to the outcomes. A body that returns passes; one that raises AssertionError is a failure; anything else is an exception, counted separately. Registered as rigor, and verified to co-exist with Opik's own pytest plugin.

The core never imports either. If a vendor SDK breaks, you lose a dashboard, not a test suite.

Designed to support SR 11-7 model validation

SR 11-7 (Federal Reserve / OCC 2011-12, April 2011) is the US supervisory guidance on model risk management. It is not a checklist and this library does not make you compliant with it — compliance is an institutional programme with independent review, governance, and validators, and no Python package delivers that.

What rigor does is make three things it asks for cheap to produce as a by-product of testing, rather than reconstructed from memory at review time:

Conceptual soundness is documented where the choices are made. Why Wilson over Clopper-Pearson (coverage vs power at the small n an eval can afford), why Mann-Whitney over a t-test (ordinal, non-normal, multi-modal scores), and why a parse failure is never a fail-verdict — all in the docstrings of the functions that implement them, not in a slide deck that drifts away from the code.

Ongoing monitoring is what the gates are. A recorded baseline carries a sha256 of its own contents and is verified on load, so a regression cannot be made to disappear by editing the file it is compared against.

Outcomes analysis is what the evidence log holds: every verdict, every sample, every gate decision, appended and never rewritten, each carrying the judge's pinned model id and the sha256 of the exact rubric revision that produced it. The question "what exactly was this number measured with, and has that changed since?" has a file-backed answer.

Effective challenge is a property of your organisation, not your tooling. But challenge needs something to bite on, and "the rubric hashed to e62bdbb2… and the judge was pinned to claude-sonnet-4-5-20250929" is a materially better starting point than "we ran the eval and it looked fine."

If you work in a regulated setting, treat this as plumbing that makes evidence falsifiable and cheap — and treat the guidance as your compliance team's to interpret.

Roadmap

0.2.0 is still two primitives done rigorously; it fixes and packages, and adds no third. These are the good ideas that were deliberately parked, most of them discovered by writing the example and finding the library annoying to use:

• A non-raising check_* beside each assert_*. Today success returns a report dict and failure carries the same numbers on exc.stats — and underpowered/runs_needed exist only on the failure path. Printing "what did the gate conclude" means try/except around every gate.
• Typed report objects. The reports are dict[str, Any]: no autocomplete, no typo protection, and the key names are not guessable (lower_bound vs interval_lower vs min_rate).
• sample_over(items, fn). sample(fn, n) hands fn nothing, so every caller writing a real eval reimplements the same dataset-cycling closure. Not to be confused with the shipped sample_of(values), which wraps values you already have; the collision of names is itself a reason this one may end up called something else.
• A seedable callable FakeAdapter. seed= is rejected in exactly the responses=<callable> mode a realistic fake needs — the one shape that can react to its input is the one that cannot take a seed.
• Cost and latency gates. SampleResult already records per-run durations.
• Clopper-Pearson as an option, for settings that need guaranteed coverage.
• A configurable score range (currently fixed at 1–5).

Explicitly not planned: becoming an eval platform. Datasets, dashboards, prompt management, and orchestration are what Opik is for, and rigor integrates with it rather than competing.

Development

Linux and macOS:

python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
.venv/bin/python -m pytest tests examples
.venv/bin/python -m ruff check src tests scripts

Windows (PowerShell), which is where this project is primarily developed — the interpreter is under Scripts\, not bin/, and it is python.exe:

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\python.exe -m pytest tests examples
.\.venv\Scripts\python.exe -m ruff check src tests scripts

pytest on its own honours testpaths = ["tests"] from pyproject.toml and does not collect examples/, so it reports fewer tests than pytest tests examples. Neither number is wrong; they are answers to different questions.

The suite is green with no credentials and no network. Anything needing a live provider is marked requires_network or requires_opik and deselected in CI. The Opik integration is tested against a real Opik client via opik.record_traces_locally() pointed at a loopback server — not against mocks.

Two environments are maintained: one without Opik (the suite must pass without it) and one with, for the integration and plugin co-installation tests.

License

MIT — see LICENSE. The full text also ships inside the wheel, under opik_rigor-<version>.dist-info/licenses/LICENSE; the link is absolute because PyPI renders this file with no repository behind it, so a relative one 404s from the project page.