DeltaCert

Certify any change to your LLM serving stack — quantization, engine upgrades, batch size, model updates — with a mathematical bound, before you deploy.

Built by Threvo Labs.

PyPI · SPEC.md · LICENSE

pip install deltacert

The catch

Standard benchmarks said this 4-bit quantized model was fine:

GSM8K said +1.0% — looks safe. DeltaCert flagged it unsafe. The generated text actually forks from the fp16 model at token 14, on long-form coding generations, well before the math bound (token 31) would even flag it. Same nf4 config, three independent signals, one real finding: a standard short-form benchmark missed a regression that shows up on longer generations.

Robustness-checked: excluding all 7 references with degenerate repetition, every statistic is identical (cert_trajectory_clean7.json).

Reproduce it yourself:

deltacert generate-cases --model meta-llama/Llama-3.1-8B-Instruct --output cases.jsonl
deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int4 \
    --checks trajectory --trajectory-cases cases.jsonl

Every number above traces to a real certificate in validation_results/; every row in the table below reproduces with one script.

The proof

Seven real flagship tests, each a full before/after comparison on a real model, real GPU, real downstream benchmark:

Five safe, two unsafe. A tool that only ever says "safe" isn't measuring anything — the two unsafe rows above are DeltaCert doing its job.

How it works

DeltaCert compares output distributions before and after a change. From the cosine similarity c between two runs, it computes:

Δ = 4c√(1-c²)        (commutator magnitude)
d = -log(Δ/2)        (algebraic distance)
divergence_bound = 2·exp(-d)

d is an algebraic distance; 2e⁻ᵈ is the certified bound on output divergence — deterministic, minutes to compute, checkable at every token position, no eval harness or labeled data required.

"Certified" throughout this document means: measured against a calibrated threshold with a stated bound — not a guarantee of downstream quality.

• Full derivation, clamp behavior, per-method calibration (with sample sizes disclosed), and the top-k logprobs caveat: see SPEC.md.
• This README asserts. The spec defends. Nothing here is a proof.

Getting started

pip install deltacert

deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int8

That uses DeltaCert's shipped reference calibration (from the 7-test suite above). For a threshold tuned to your own model and workload, run the sweep yourself:

deltacert capture --model your-model --output baseline.npz
deltacert capture --model your-model --quantization int8 --output candidate.npz
deltacert calibrate --baseline baseline.npz --candidates candidate.npz \
    --names int8 --downstream-file your_evals.json

deltacert certify always tells you when it's using the shipped calibration instead of your own.

Integrations

• vLLM plugin — official vllm.general_plugins entry point, already wired in this package. Opt-in only: complete no-op unless DELTACERT_ENFORCE=1 is set, so pip install deltacert is safe in a shared image; when enabled, a serving engine refuses to start on an uncertified change.
• CI/CD gate — python -m deltacert.integrations.cicd_hook --cert ./cert.json exits 1 (blocks the pipeline) if not certified, 0 if certified. Works with GitHub Actions, GitLab CI, Jenkins, or any CI that checks exit codes.
• HuggingFace auto-wiring — from deltacert.integrations.hf_integration import auto_certify picks the right collectors for you from what's active in your config (quantization, LoRA, prefix cache) instead of calling certify_system() with raw parameters yourself.

Verifying a certificate

Optional — certificates work unsigned; signing adds tamper-evidence for sharing certs across teams or with auditors.

deltacert keygen --private-key mykey.pem --public-key mykey.pub
deltacert sign --cert cert.json --key-file mykey.pem
deltacert verify --cert cert.json --key-file mykey.pub

verify exits 0 if the signature is valid, 1 if the certificate was modified after signing or signed by a different key. Every certificate also carries a validation_status field (flagship_validated vs implemented_pending_validation) — signed as part of the payload, so a signature can never make an unvalidated collector's result look more trustworthy than it is.

Keep your private key secret — never commit it, never share it. Only the public key is meant to be distributed.

All 13 reference certificates in validation_results/ are signed with Threvo's key (deltacert-public.pem, committed in this repo). Don't take our numbers on faith — check them yourself:

deltacert verify --cert validation_results/weight_quant/cert_nf4.json --key-file deltacert-public.pem

What it certifies

DeltaCert ships all 20 collectors described in the design — the code is real and implemented, and doesn't get deleted just because a given check hasn't been run in a full end-to-end validation yet. Only 7 have real flagship validation results behind them so far (the table above); the rest are working code with the same math, not yet run through that process.

"Python API only" means the check needs code you supply (a custom compress_fn, attention mask, etc.) — the CLI can't conjure that for you; see import deltacert as dc; dc.certify_system(...).

Limitations (stated up front, not discovered by you later)

• The 7 validated results above all use one model family (Llama-3.1-8B-Instruct) plus one hosted API (gpt-4o-mini). Cross-model generalization isn't proven yet.
• Per-method calibration (e.g. the bnb/GPTQ thresholds) is an initial calibration from n=5 configs on one model — not a settled constant. Run deltacert calibrate on your own model/workload rather than trusting the shipped default for anything production-critical.
• The provider_drift result above is a same-day proxy (pinned snapshot vs. current alias), not the real weekly-cadence drift measurement, which needs two runs across real time.
• d_comm is a reliable within-method damage indicator but is not directly comparable across different compression methods — see SPEC.md for the bnb-vs-GPTQ false-negative this caused and how it's handled.

Roadmap

• More models, more downstream tasks — firming up per-method calibration beyond n=5
• Full validation pass on the remaining 13 collectors
• Real weekly-cadence provider_drift run (beyond the same-day proxy)
• Cross-backend certification: extend capture to TensorRT-LLM / SGLang (comparison logic is already backend-agnostic)

Citation

If you use DeltaCert, cite it as:

@software{deltacert2026,
  title  = {DeltaCert: Calibrated Divergence Certification for LLM Serving Systems},
  author = {Shorya},
  year   = {2026},
  url    = {https://pypi.org/project/deltacert/}
}

License

Apache-2.0. See LICENSE.

Contact

• Issues and questions: open a GitHub issue
• Full validation data: validation_results/ in this repo