invarlock 0.16.0

<img
  src="https://raw.githubusercontent.com/invarlock/invarlock/v0.16.0/docs/assets/invarlock-logo.svg"
  alt="InvarLock"
  width="420"
/>

Evaluate model changes. Verify the evidence. Share the result.

InvarLock evaluates model changes and produces evidence that another team can verify independently. Compare a candidate (the subject) with an approved baseline, using tests and acceptance thresholds you choose. Run a supported comparison through InvarLock, or bring per-case records from your existing evaluation workflow.

The result is a signed evidence bundle, a separate verification receipt and a report explaining the comparison. Your customer or internal reviewer can check the retained result against agreed inputs and policy without rerunning model inference.

• Three native scorers: exact match, normalized NLL and bounded LLM judging.
• Existing workflows: supported evaluator exports and a Python capture SDK.
• Offline verification: replay the supported analysis with recipient-owned trust inputs; no provider credentials needed.
• Reviewable results: model and service identities, measured changes, uncertainty bounds and the checks that passed or failed.

Quickstart

Try a signed evidence check on a regular CPU. Python 3.12+ is required; no GPU, model download, API key or container engine is needed for this example.

From a checkout of this repository:

python -m venv .venv
. .venv/bin/activate
python -m pip install .
python examples/quickstart/run.py \
  --fixture examples/acceptance-handoff/golden

The command verifies retained evidence against the example's separate trust inputs, issues a new signed receipt and writes an HTML report. It prints:

PASS signed evidence verified
Decision: pass

Open invarlock-quickstart-output/evidence.html. The same directory contains verification.receipt.json and verification.result.json. This demonstrates verification and reporting of a fixed comparison; it makes no new model calls.

python -m venv .venv
. .venv/bin/activate
python -m pip install invarlock
INVARLOCK_VERSION="$(python -c 'from importlib.metadata import version; print(version("invarlock"))')"

curl -fsSLO \
  "https://github.com/invarlock/invarlock/archive/refs/tags/v${INVARLOCK_VERSION}.tar.gz" &&
tar -xzf "v${INVARLOCK_VERSION}.tar.gz" --strip-components=3 \
  "invarlock-${INVARLOCK_VERSION}/examples/quickstart" \
  "invarlock-${INVARLOCK_VERSION}/examples/acceptance-handoff/golden" &&

python run.py --fixture golden

Use documentation and examples from the same release as a published wheel. For local builds, use the exact source checkout that built the package; retain its commit and wheel digest when sharing it. A missing release archive is an error, never a reason to substitute another version. See matching wheels and examples.

What can you use it for?

These workflows fit teams that repeatedly need to produce, check or retain evidence supporting a model change: model suppliers, fine-tuning and optimization teams, and internal AI teams with a release-review process.

One workflow: evaluate, verify, report

For a captured deterministic comparison, prepare a request, signing key and independent trust profile:

invarlock evaluate request.yaml --signing-key signing-key.pem --preflight --json
invarlock evaluate request.yaml --signing-key signing-key.pem
invarlock verify evidence/ --trust-profile trust/trust-inputs.json \
  --receipt verification.receipt.json
invarlock report evidence/ --html report.html

Use the evidence destination declared in your request. The captured-results guide provides complete setup instructions, keys and trust-profile preparation. Native runs also need runtime resources; judge receipt issuance uses its own verifier-key and identity options. Follow the linked guide for that workflow.

The HTML report supports multiple metrics and slices, with an overview and per-result detail. Terminal and Markdown outputs support review; JSON and JUnit support automation where the selected workflow provides them. See the CLI reference for output options and policy exit codes.

Choose a scorer

The same native and captured entry points support three built-in scorers, each with its own required observations and statistical treatment:

Policy uses the conservative uncertainty bound, with the configured sample, precision and quality requirements. A point estimate alone does not decide the result. Schedule and policy explains the statistical scope and thresholds.

Judge collection uses the optional invarlock-inspect-judge[inspect] package with explicit call, token, cost, timeout and checkpoint limits. Importing retained ratings, verification and reporting work offline in the core wheel. Start with invarlock evaluate --init my-judge --example native-judge and replace the illustrative model pins with your actual runtime inputs. See judge scoring for the supported rubric, reference and collection profiles.

Task-specific deterministic extensions cover normalized labels, numeric tolerances, structured fields and token overlap. Evidence sets can require multiple complementary checks on the same frozen answers while preserving each check's meaning.

Keep your evaluator or run the comparison here

Use existing records. Installed parsers cover selected Inspect AI, LM Evaluation Harness and Promptfoo export profiles, plus canonical JSON and generic JSONL. The capture SDK lets other pipelines supply the required per-case facts. Structured inputs need an explicit text projection for judging; normalized NLL needs actual likelihood measurements. An aggregate score cannot substitute for missing observations. The qualification matrix distinguishes installed support, replay authority and retained runtime evidence for each declared profile.

Run a native comparison. Hugging Face Transformers is the built-in runtime. Optional GGUF/llama.cpp, TensorRT-LLM and Hugging Face vision-text packages provide additional runtime profiles. Native run mode uses a caller-authorized, digest-addressed Docker or Podman image. Follow the getting-started guide for artifact pins, runtime setup and independent verification inputs. Host-side model preparation uses the matching checkout's HF runtime group after bootstrapping its verified hardened Accelerate wheel. The import request uses complete retained provider sidecars and omits --runtime-image and --runtime-image-digest.

Recheck a hosted service. Your harness records fresh executions, service configuration and observation windows. InvarLock compares those captured facts and verifies the resulting evidence offline. Hosted identity describes the observed service; it does not claim access to hidden model weights. Your scheduler initiates periodic comparisons. Verifying old evidence does not measure the service again.

Inspect real retained examples

The repository includes signed evidence, receipts and replay instructions from actual model runs. Each reference establishes its declared workflow and scope:

A comparative pass is not proof of adequate task quality or representative production performance. The local HTTP example does not qualify an external provider; repeated judge ratings are not additional independent cases.

What verification establishes

A recipient supplies its expected identities, policy and signer trust through a channel independent of the submitted bundle. InvarLock checks the package and reconstructs the supported analysis against those expectations. Native runtime bindings and imported observations retain different provenance claims.

Verification checks retained evidence; it does not independently rerun the original model execution. It does not establish that the chosen benchmark covers production traffic, that a rubric captures every requirement, or that a passing comparison authorizes deployment. Read the assurance case and trust model for the precise guarantees and assumptions.

For artifact-delivery automation, optional acceptance attestations and OPA/Rego or CUE policies consume the authenticated result under recipient-controlled policy. v0.13 evidence and receipts remain verifiable and ingestible; acceptance always uses the recipient's current policy. InvarLock is pre-1.0, with explicit artifact format versions and a Python API that may evolve between minor releases.

Documentation and contributing

• Getting started · Examples · CLI · Python API
• Contributing for development setup and required checks.
• Discussions for questions and integration ideas; Issues for reproducible bugs.
• Security policy for private vulnerability reports.

Apache-2.0. See LICENSE and third-party notices for dependency and retained-data terms.