deltacert 1.0.4

Calibrated divergence certification for LLM serving systems

DeltaCert

Certify any change to your LLM serving stack — quantization, engine upgrades, batch size, model updates — with a mathematical bound, before you deploy.

PyPI · SPEC.md · LICENSE

pip install deltacert

---

The catch

Standard benchmarks said this 4-bit quantized model was fine:

Config	Short eval said	d_COMM	safe_until_token	failure_after_token	Verdict
Llama-3.1-8B fp16 → nf4	GSM8K 5-shot exact-match +1.0% (looks safe)	0.00	31	14	unsafe

GSM8K said +1.0% — looks safe. DeltaCert flagged it unsafe. The generated text actually forks from the fp16 model at token 14, on long-form coding generations, well before the math bound (token 31) would even flag it. Same nf4 config, three independent signals, one real finding: a standard short-form benchmark missed a regression that shows up on longer generations.

Reproduce it yourself:

deltacert generate-cases --model meta-llama/Llama-3.1-8B-Instruct --output cases.jsonl
deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int4 \
    --checks trajectory --trajectory-cases cases.jsonl

Every number above traces to a real certificate in validation_results/; every row in the table below reproduces with one script.

The proof

Seven real flagship tests, each a full before/after comparison on a real model, real GPU, real downstream benchmark:

Change	Business gain	d_COMM	Downstream effect	Verdict
Llama-3.1-8B batch=1 → batch=64	64 concurrent requests, same GPU	6.21	GSM8K -1.0 pt	✅ Safe
vLLM 0.8.5 → vLLM 0.9.0	take the upgrade same-week, not months later	16.09	GSM8K -1.0 pt	✅ Safe
KV cache default → fp8 (vLLM native)	2x concurrent capacity	4.83	GSM8K -1.0 pt	✅ Safe
gpt-4o-mini pinned snapshot → current alias	same-day provider-drift check	6.65	canary acc +0.0 pt	✅ Safe
Standard decode → speculative decode (k=5)	claimed ~2x throughput	15.38	GSM8K +0.0 pt, measured 0.28x (slower)	✅ Safe on quality, not on speed
Llama-3.1-8B fp16 → nf4 (W4)	+60% VRAM reduction	0.00	forks at token 14 on long generations	❌ Unsafe
Llama-3.1-8B fp16 → GPTQ int4	+75% VRAM reduction	1.16	GSM8K -8.0 pts	❌ Unsafe

Five safe, two unsafe. A tool that only ever says "safe" isn't measuring anything — the two unsafe rows above are DeltaCert doing its job.

How it works

DeltaCert compares output distributions before and after a change. From the cosine similarity c between two runs, it computes:

Δ = 4c√(1-c²)        (commutator magnitude)
d = -log(Δ/2)        (algebraic distance)
divergence_bound = 2·exp(-d)

d is an algebraic distance; 2e⁻ᵈ is the certified bound on output divergence — deterministic, minutes to compute, checkable at every token position, no eval harness or labeled data required.

"Certified" throughout this document means: measured against a calibrated threshold with a stated bound — not a guarantee of downstream quality.

• Full derivation, clamp behavior, per-method calibration (with sample sizes disclosed), and the top-k logprobs caveat: see SPEC.md.
• This README asserts. The spec defends. Nothing here is a proof.

Getting started

pip install deltacert

deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int8

That uses DeltaCert's shipped reference calibration (from the 7-test suite above). For a threshold tuned to your own model and workload, run the sweep yourself:

deltacert capture --model your-model --output baseline.npz
deltacert capture --model your-model --quantization int8 --output candidate.npz
deltacert calibrate --baseline baseline.npz --candidates candidate.npz \
    --names int8 --downstream-file your_evals.json

deltacert certify always tells you when it's using the shipped calibration instead of your own.

What it certifies

DeltaCert ships all 20 collectors described in the design — the code is real, tested, and doesn't get deleted just because a given check hasn't been run in a full end-to-end validation yet. Only 7 have real flagship validation results behind them so far (the table above); the rest are working code with the same math, not yet run through that process.

#	Check	CLI-drivable	Status
1	weight_quant	✅	✅ validated
2	kv_cache_quant	✅	✅ validated
3	batch_divergence	✅ (needs vLLM)	✅ validated
4	spec_decoding	✅ (needs vLLM)	✅ validated
5	engine_swap	✅	✅ validated
6	provider_drift	✅	✅ validated
7	trajectory	✅	✅ validated
8	activation_quant	✅	🔬 code exists, not yet validated
9	prefix_cache	✅	🔬 code exists, not yet validated
10	lora	✅	🔬 code exists, not yet validated
11	model_swap	✅	🔬 code exists, not yet validated
12	prompt_swap	✅	🔬 code exists, not yet validated
13	sparse_attention	Python API only	🔬 code exists, not yet validated
14	moe_token_dropping	Python API only	🔬 code exists, not yet validated
15	neuron_skipping	Python API only	🔬 code exists, not yet validated
16	allreduce_tp	Python API only	🔬 code exists, not yet validated
17	alltoall_ep	Python API only	🔬 code exists, not yet validated
18	pipeline_parallel	Python API only	🔬 code exists, not yet validated
19	kv_transfer	Python API only	🔬 code exists, not yet validated
20	gradient_compress	Python API only	🔬 code exists, not yet validated

"Python API only" means the check needs code you supply (a custom compress_fn, attention mask, etc.) — the CLI can't conjure that for you; see import deltacert as dc; dc.certify_system(...).

Limitations (stated up front, not discovered by you later)

• The 7 validated results above all use one model family (Llama-3.1-8B-Instruct) plus one hosted API (gpt-4o-mini). Cross-model generalization isn't proven yet.
• Per-method calibration (e.g. the bnb/GPTQ thresholds) is an initial calibration from n=5 configs on one model — not a settled constant. Run deltacert calibrate on your own model/workload rather than trusting the shipped default for anything production-critical.
• The provider_drift result above is a same-day proxy (pinned snapshot vs. current alias), not the real weekly-cadence drift measurement, which needs two runs across real time.
• d_comm is a reliable within-method damage indicator but is not directly comparable across different compression methods — see SPEC.md for the bnb-vs-GPTQ false-negative this caused and how it's handled.

Roadmap

• More models, more downstream tasks — firming up per-method calibration beyond n=5
• Full validation pass on the remaining 13 collectors
• Real weekly-cadence provider_drift run (beyond the same-day proxy)

Citation

If you use DeltaCert, cite it as:

@software{deltacert2026,
  title  = {DeltaCert: Calibrated Divergence Certification for LLM Serving Systems},
  author = {Shorya},
  year   = {2026},
  url    = {https://pypi.org/project/deltacert/}
}

License

Apache-2.0. See LICENSE.

Contact

• Issues and questions: open a GitHub issue
• Full validation data: validation_results/ in this repo