regression-substrate 0.1.0

A statistically rigorous CI gate for AI: treats model outputs as distributions, penalizes unreliable judges, and decides ship / hold / regression.

regression-substrate

A statistically rigorous CI gate for AI systems. It treats model outputs as distributions, penalizes unreliable judges, and returns a SHIP / HOLD / REGRESSION verdict you can block a pull request on.

Install

pip install regression-substrate            # core (numpy, scipy)
pip install "regression-substrate[clustering]"   # + auto_cluster (scikit-learn)
pip install "regression-substrate[langsmith]"    # + LangSmith adapter

For development (editable install with test dependencies):

git clone <repo-url>
cd regression-substrate
pip install -e ".[dev]"

CLI (drop into CI)

regsub --data evals.csv --gold gold.jsonl --version-a v1 --version-b v2 --out out/
# exit 0 = SHIP / SHIP_WITH_FLAGS ; 1 = REGRESSION / HOLD ; 2 = JUDGE_INADMISSIBLE

One line in your CI pipeline blocks the PR on a regression.

Library

from regression_substrate import gate, load_from_csv, Judge

judge = Judge(my_llm_scorer)            # any (input, response) -> [0,1]
cal = judge.calibrate(gold_records)     # -> kappa, error_sd
sa, sb, cids, meta = load_from_csv("evals.csv", "v1", "v2")
decision = gate(sa, sb, cids, judge_error_sd=cal["error_sd"], kappa=cal["kappa"])
print(decision.verdict)

What's inside

Module	Purpose
diff_engine	Offline gate: variance components, bootstrap CI, cluster scan, BH/e-BH
ingest	Loaders (JSONL, CSV), judge harness, auto-clustering
sequential_gate	Always-valid martingale monitor for continuous deployment
gold	Rolling gold set, drift detection, forced sampling for labeling
adapters	Vendor flatteners (LangSmith preset)
otel_exporter	OTel-aligned span capture path
cli	The regsub console command

Running tests

pip install -e ".[dev]"
pytest

See examples/ for a runnable dataset and CHANGES.md for design decisions.