regression-substrate 0.2.0

A statistically rigorous CI gate for AI: treats model outputs as distributions, penalizes unreliable judges, and decides ship / hold / regression.

regression-substrate

A statistically rigorous CI gate for AI systems. It treats model outputs as distributions, penalizes unreliable judges, and returns a SHIP / HOLD / REGRESSION verdict you can block a pull request on.

Install

pip install regression-substrate            # core (numpy, scipy)
pip install "regression-substrate[clustering]"   # + auto_cluster (scikit-learn)
pip install "regression-substrate[langsmith]"    # + LangSmith adapter

For development (editable install with test dependencies):

git clone <repo-url>
cd regression-substrate
pip install -e ".[dev]"

CLI (drop into CI)

regsub --data evals.csv --gold gold.jsonl --version-a v1 --version-b v2 --out out/
# exit 0 = SHIP / SHIP_WITH_FLAGS ; 1 = REGRESSION / HOLD ; 2 = JUDGE_INADMISSIBLE

One line in your CI pipeline blocks the PR on a regression.

Library

from regression_substrate import gate, load_from_csv, Judge

judge = Judge(my_llm_scorer)            # any (input, response) -> [0,1]
cal = judge.calibrate(gold_records)     # -> kappa, error_sd
sa, sb, cids, meta = load_from_csv("evals.csv", "v1", "v2")
decision = gate(sa, sb, cids, judge_error_sd=cal["error_sd"], kappa=cal["kappa"])
print(decision.verdict)

"I have a chatbot and I changed the prompt — now what?"

The library does the statistics for free. The only work on your side is producing scores. The fastest path:

pip install regression-substrate
python -m regression_substrate.template     # writes eval_template.py

Open eval_template.py and fill in three blanks — your app function, your judge, and your test questions:

from regression_substrate import LLMJudge
import openai
score = LLMJudge.from_openai(openai.OpenAI())   # built-in judge, no scoring code to write

def run_app(question, version):
    return my_chatbot(question, prompt=PROMPTS[version])

QUESTIONS = ["How do I get a refund?", "What are your hours?", ...]   # 30+ for a real verdict

Run it (python eval_template.py) and you get a SHIP / HOLD / REGRESSION verdict. That's ~15 lines of glue, not 300.

Built-in LLM judge

You don't have to write a scorer. LLMJudge wraps any provider:

from regression_substrate import LLMJudge, Judge

score = LLMJudge.from_openai(client)          # or .from_anthropic(client)
# or fully provider-agnostic — pass any complete(prompt)->str:
score = LLMJudge(lambda prompt: my_llm(prompt))

cal = Judge(score).calibrate(gold_records)    # measures kappa + error_sd

Use temperature 0 so scores are stable when you replay. Then still calibrate it against ~20 hand-labeled examples — an uncalibrated judge can't be trusted to gate.

The four things every project provides

The gate, ingestion, calibration, and clustering are free. What you supply is project-specific: (1) eval data — a CSV of input,version,score; (2) a judge — now mostly covered by LLMJudge; (3) a small gold set — ~20 hand-labeled rows; and (4) a way to run the same inputs through both versions. The template scaffolds (1) and (4) and wires in (2) and (3).

What's inside

Module	Purpose
diff_engine	Offline gate: variance components, bootstrap CI, cluster scan, BH/e-BH
ingest	Loaders (JSONL, CSV), judge harness, auto-clustering
judges	LLMJudge — provider-agnostic LLM-as-judge (OpenAI, Anthropic, Groq, custom)
template	write_template() — scaffold a fill-in-the-blanks eval script
sequential_gate	Always-valid martingale monitor for continuous deployment
gold	Rolling gold set, drift detection, forced sampling for labeling
adapters	Vendor flatteners (LangSmith preset)
otel_exporter	OTel-aligned span capture path
cli	The regsub console command

Running tests

pip install -e ".[dev]"
pytest

See examples/ for a runnable dataset and CHANGES.md for design decisions.