regression-substrate 0.3.0

A statistically rigorous CI gate for AI: treats model outputs as distributions, penalizes unreliable judges, and decides ship / hold / regression.

regression-substrate

A statistically rigorous CI gate for AI systems. It treats model outputs as distributions, penalizes unreliable judges, and returns a SHIP / HOLD / REGRESSION verdict you can block a pull request on.

Install

pip install regression-substrate            # core (numpy, scipy)
pip install "regression-substrate[clustering]"   # + auto_cluster (scikit-learn)
pip install "regression-substrate[langsmith]"    # + LangSmith adapter

For development (editable install with test dependencies):

git clone <repo-url>
cd regression-substrate
pip install -e ".[dev]"

CLI (drop into CI)

regsub --data evals.csv --gold gold.jsonl --version-a v1 --version-b v2 --out out/
# exit 0 = SHIP / SHIP_WITH_FLAGS ; 1 = REGRESSION / HOLD ; 2 = JUDGE_INADMISSIBLE

One line in your CI pipeline blocks the PR on a regression.

Library

from regression_substrate import gate, load_from_csv, Judge

judge = Judge(my_llm_scorer)            # any (input, response) -> [0,1]
cal = judge.calibrate(gold_records)     # -> kappa, error_sd
sa, sb, cids, meta = load_from_csv("evals.csv", "v1", "v2")
decision = gate(sa, sb, cids, judge_error_sd=cal["error_sd"], kappa=cal["kappa"])
print(decision.verdict)

"I have a chatbot and I changed the prompt — now what?"

The library does the statistics for free. The only work on your side is producing scores. The fastest path:

pip install regression-substrate
python -m regression_substrate.template     # writes eval_template.py

Open eval_template.py and fill in three blanks — your app function, your judge, and your test questions:

from regression_substrate import LLMJudge
import openai
score = LLMJudge.from_openai(openai.OpenAI())   # built-in judge, no scoring code to write

def run_app(question, version):
    return my_chatbot(question, prompt=PROMPTS[version])

QUESTIONS = ["How do I get a refund?", "What are your hours?", ...]   # 30+ for a real verdict

Run it (python eval_template.py) and you get a SHIP / HOLD / REGRESSION verdict. That's ~15 lines of glue, not 300.

Built-in LLM judge

You don't have to write a scorer. LLMJudge wraps any provider:

from regression_substrate import LLMJudge, Judge

score = LLMJudge.from_openai(client)          # or .from_anthropic(client)
# or fully provider-agnostic — pass any complete(prompt)->str:
score = LLMJudge(lambda prompt: my_llm(prompt))

cal = Judge(score).calibrate(gold_records)    # measures kappa + error_sd

Use temperature 0 so scores are stable when you replay. Then still calibrate it against ~20 hand-labeled examples — an uncalibrated judge can't be trusted to gate.

The four things every project provides

The gate, ingestion, calibration, and clustering are free. What you supply is project-specific: (1) eval data — a CSV of input,version,score; (2) a judge — now mostly covered by LLMJudge; (3) a small gold set — ~20 hand-labeled rows; and (4) a way to run the same inputs through both versions. The template scaffolds (1) and (4) and wires in (2) and (3).

Calibration guide (read before trusting any verdict)

The gate's power comes from penalizing unreliable judges — which only works if calibration is honest. The rules:

1. The gold set must be independent human judgment. Label ~20–50 examples by hand. If the human labels come from the same rule your judge applies, kappa and error_sd are circular and meaningless (you'll get a fake kappa=1.0, error_sd=0 — the library warns when it sees this pattern).
2. kappa >= 0.4 to gate. Below that, gate() returns JUDGE_INADMISSIBLE — that refusal is protecting you.
3. Format: each gold record is {"input": ..., "response": ..., "human": <your 0-1 score>, "reference": <optional>}.
4. error_sd is propagated, not cosmetic — a noisier judge widens the CI and makes SHIP harder. That's the design working.

Structured outputs (SQL, code, JSON): a worked example

LLMJudge is for prose. For structured outputs, build a reference scorer from the building blocks in regression_substrate.scorers — here's a SQL structural scorer in ~10 lines:

import re
from regression_substrate.scorers import set_overlap

def sql_structural_score(question, answer, reference=None):
    if not reference:
        return 0.0
    ident = lambda sql: set(re.findall(r"[A-Za-z_][A-Za-z0-9_]*", sql.lower())) \
                        - {"select", "from", "where", "join", "on", "and", "or",
                           "group", "by", "order", "limit", "as"}
    return set_overlap(ident(answer), ident(reference))

These lexical scorers are weak proxies — identifier overlap scores non-executing SQL above a correct rewrite. Calibrate against independent human labels before gating on one; the kappa check exists for exactly this case.

What's inside

Module	Purpose
diff_engine	Offline gate: variance components, bootstrap CI, cluster scan, BH/e-BH
ingest	Loaders (JSONL, CSV), judge harness, auto-clustering
judges	LLMJudge — provider-agnostic LLM-as-judge (OpenAI, Anthropic, Groq, custom)
scorers	Reference-based building blocks: exact_match, jaccard, token_f1, set_overlap, json_overlap
template	write_template() — scaffold a fill-in-the-blanks eval script
sequential_gate	Always-valid martingale monitor for continuous deployment
gold	Rolling gold set, drift detection, forced sampling for labeling
adapters	Vendor flatteners (LangSmith preset)
otel_exporter	OTel-aligned span capture path
cli	The regsub console command

Running tests

pip install -e ".[dev]"
pytest

See examples/ for a runnable dataset and CHANGES.md for design decisions.