invarlock 0.9.0

Edit‑agnostic robustness evaluation reports for weight edits (InvarLock framework)

Edit‑agnostic robustness reports for weight edits

Catch silent quality regressions from quantization, pruning, and weight edits before they ship.

Quantizing, pruning, or otherwise editing a model’s weights can silently degrade quality. InvarLock compares an edited subject checkpoint against a fixed baseline with paired evaluation windows, enforces the canonical guard chain (invariants -> spectral -> RMT -> variance -> invariants), and produces a machine-readable evaluation report you can gate in CI.

Why InvarLock?

• Quality gates for edited checkpoints: catch regressions before deployment.
• Paired statistical evidence: primary metrics with confidence intervals.
• Auditable evidence: deterministic pairing metadata + policy digests in evaluation.report.json.
• CI/CD-friendly: stable exit codes, --json outputs, and portable “evidence packs”.
• Offline-first: network is disabled by default; enable downloads per command.
• Explicit assurance boundary: the trust model states what a strict pass means and what it does not mean.

Who is this for?

• ML engineers shipping edited model checkpoints, including quantized, pruned, fine-tuned, or otherwise weight-modified variants.
• MLOps and platform teams building CI gates, runtime-provenance verification, and reviewable evaluation artifacts.
• Researchers validating weight-edit, compression, and model-comparison methods with reproducible paired evaluation across text and image-text workflows supported here.

How it works

┌───────────────────────┐     ┌────────────────────────────────────────────┐
│ Baseline (checkpoint) │────►│                                            │
└───────────────────────┘     │  invarlock evaluate                        │
                              │  ├─► Paired windows (deterministic)        │
┌───────────────────────┐     │  ├─► GuardChain pipeline                   │
│ Subject  (checkpoint) │────►│  │   └─► invariants -> spectral -> RMT -> VE │
└───────────────────────┘     │  └─► Emit: evaluation.report.json          │
                              │                                            │
                              └────────────────────────────────────────────┘
                                                     │
                                     ┌───────────────┴───────────────┐
                                     ▼                               ▼
                                 ✅ PASS                          ❌ FAIL
                                 (ship)                          (rollback)

Quick Start

Colab (CPU-friendly):

The public front door is evaluate -> verify -> report html, but the repo now splits onboarding by user type:

• Wheel user / reviewer: install invarlock, inspect an existing evaluation.report.json, and render HTML without cloning the repository.
• Evaluator: install invarlock[hf] when you want evaluate to load Hugging Face models and emit a fresh evaluation bundle.
• Repo maintainer: clone the repo and build the local runtime image when you need maintainer smokes, repo presets, or local container-image iteration.

The default evaluate path runs model-loading commands inside the runtime container and expects an OCI engine such as podman or docker. Host-side workflows can opt into --execution-mode host, but the default verification path below expects a container-backed report with sibling runtime provenance. evaluate also defaults to the current strict assurance contract: CI/release profile, balanced/conservative tier, canonical guard order, complete evidence, strict paired metrics, and verified runtime provenance are required for an assurance pass. Use --assurance off only for exploratory reports.

# Evaluator path: create a fresh bundle
pip install "invarlock[hf]"

invarlock --version

# Compare baseline vs subject (downloads require explicit network enable)
invarlock evaluate --allow-network \
  --baseline gpt2 \
  --subject  distilgpt2 \
  --adapter auto \
  --profile ci \
  --assurance strict \
  --report-out reports/eval \
  --quiet

# Validate the container-backed evaluation report
test -f reports/eval/runtime.manifest.json
invarlock verify --json reports/eval/evaluation.report.json

# Render HTML for sharing
invarlock report html -i reports/eval/evaluation.report.json -o reports/eval/evaluation.html

Wheel-only review path: pip install invarlock, invarlock doctor, invarlock verify /path/to/evaluation.report.json, and invarlock report html -i /path/to/evaluation.report.json -o /path/to/evaluation.html.

Repo maintainers can build the local runtime image once with make runtime-image; InvarLock automatically prefers invarlock-runtime:local when it is present.

Artifact model:

Artifact	Produced by	Primary consumers
evaluation.report.json	invarlock evaluate, invarlock report generate --format report	invarlock verify, invarlock report html, invarlock report validate, invarlock report explain --evaluation-report, invarlock advanced runtime-verify
report.json	Baseline/subject run directories under runs/...	invarlock report generate, invarlock report explain --subject-report ... --baseline-report ...

Example output (abridged; counts vary by profile/config):

INVARLOCK v<version> · EVALUATE
Baseline: gpt2 -> Subject: gpt2 · Profile: dev
Status: PASS · Gates: <passed>/<total> passed
Primary metric ratio: <ratio>
Output: reports/eval/evaluation.report.json
Runtime provenance: reports/eval/runtime.manifest.json

Command Surface

• First touch in a fresh install: invarlock --help, invarlock --version, invarlock report --help, and invarlock advanced --help.
• Core workflow: invarlock evaluate -> invarlock verify -> invarlock report html.
• Follow-on report analysis after the core loop: invarlock report generate, invarlock report explain, and invarlock report validate.
• Environment and release checks: invarlock doctor plus the JSON surfaces emitted by doctor --json and advanced plugins ... --json.
• Runtime-manifest verifier: invarlock advanced runtime-verify --report <evaluation.report.json> --manifest <runtime.manifest.json>.
• The public contract catalog exposed by those JSON surfaces includes validation_keys, console_labels, and metric_kinds.
• Advanced workflows: invarlock advanced evidence-pack, invarlock advanced policy, invarlock advanced plugins, and invarlock advanced calibrate.
• Host execution for the core evaluate path uses --execution-mode host.
• Optional adapter/backend installs use normal Python extras such as pip install "invarlock[hf]" rather than CLI install commands.

Evidence packs (portable evidence bundles)

Evidence packs bundle reports + verification metadata into a distributable artifact.

• Guide: https://invarlock.github.io/invarlock/0.9.0/user-guide/evidence-packs/
• Verify from an installed wheel: invarlock advanced evidence-pack verify <dir> --strict
• Repo harness alternative: scripts/evidence_packs/verify_pack.sh --pack <dir> --strict

Note: configs/ and most scripts/ remain repo resources and are not included in wheels. Installed wheels include the public contracts and the invarlock advanced evidence-pack verify verifier, so installed packages can check bundles without cloning the repository.

Installation

# Minimal CLI (no torch/transformers)
pip install invarlock

# HF workflows (torch/transformers)
pip install "invarlock[hf]"

Optional extras: invarlock[probes], invarlock[gpu], invarlock[awq,gptq]. On Python 3.13+ stacks, gptq may still require a vendor wheel or a supported older interpreter because upstream auto-gptq packaging is narrower than the core InvarLock support matrix. Full setup: https://invarlock.github.io/invarlock/0.9.0/user-guide/getting-started/.

The minimal install covers the core verification and reporting flows. Add invarlock[hf] only for model-loading evaluate runs, and use the installed wheel's evidence-pack verifier when you need to inspect a bundle without cloning the repository.

Documentation

• Docs home: https://invarlock.github.io/invarlock/0.9.0/
• Quickstart: https://invarlock.github.io/invarlock/0.9.0/user-guide/quickstart/
• Compare & evaluate (BYOE): https://invarlock.github.io/invarlock/0.9.0/user-guide/compare-and-evaluate/
• Reading a report: https://invarlock.github.io/invarlock/0.9.0/user-guide/reading-report/
• CLI reference: https://invarlock.github.io/invarlock/0.9.0/reference/cli/
• Assurance case: https://invarlock.github.io/invarlock/0.9.0/assurance/00-assurance-case/ (repo source: docs/assurance/00-assurance-case.md)
• Threat model: https://invarlock.github.io/invarlock/0.9.0/security/threat-model/

Community

• Questions/ideas: https://github.com/invarlock/invarlock/discussions
• Bug reports: https://github.com/invarlock/invarlock/issues
• Contact: mailto:[e-mail]

Citation

If you use InvarLock in scientific work, please cite it (canonical metadata is in CITATION.cff):

@software{invarlock,
  title  = {InvarLock: Edit-agnostic robustness evaluation reports for weight edits},
  author = {{InvarLock}},
  url    = {https://github.com/invarlock/invarlock},
}

Limitations

• InvarLock evaluates an edited model relative to a baseline under a specific configuration; results are not “global” guarantees.
• Not a content-safety/alignment tool.
• Native Windows is not supported (use WSL2 or Linux).

Support matrix

Platform	Status	Notes
Python 3.12+	✅ Required
Linux	✅ Full	Primary dev target
macOS (Intel/M-series)	✅ Full	MPS supported (default on Apple Silicon)
Windows	❌ Not supported	Use WSL2 or a Linux container if required
CUDA	✅ Recommended	For larger models
CPU	✅ Fallback	Slower but functional

Project status

InvarLock is pre‑1.0. Until 1.0, minor releases may include breaking changes. See CHANGELOG.md.

For guidance on where to ask questions, how to report bugs, and what to expect in terms of response times, see SUPPORT.md.

Contributing

• Contributing guide: https://github.com/invarlock/invarlock/blob/v0.9.0/CONTRIBUTING.md
• Fast local checks (repo clone):
  • make targets auto-select Python 3.12+, preferring an active 3.12 env, python3.12, then the Conda env invarlock-py312 when present.
  • make dev-install
  • make test
  • make lint
  • make docs-live

License

Apache-2.0 — see LICENSE.