Conformance

Verify what your AI agent did — not what it said.

Conformance injects deliberate faults at your agent's effect boundary (a redelivered queue event, a crash mid-write, a withheld approval, a raced turn), records what actually happened, and evaluates nine deterministic invariants over that record. No language model judges anything. Every finding comes with a content-addressed witness a sceptic can re-derive offline, without you.

$ conformance verify --scenario shipbot --seed 17

seed 17 · spec conformance-1.0 · 8 schedule(s) run

✕ 1 custody                  VIOLATED    wit_05b69e04ec177d5209056ab82d
⚠ 2 one-advance              inconclusive (no attachment point for staged_race…)
⚠ 3 no-scarce-hold           inconclusive (no pool acquisitions in record…)
✕ 4 quarantine               VIOLATED    wit_5fde8bf97de2a976d8a57fbd21
✕ 5 exactly-once-effect      VIOLATED    wit_d223f1f2cfec1b7eca4e89552a
⚠ 6 plan-identity            inconclusive (needs its own recording…)
⚠ 7 timeout-equals-cancel    inconclusive (no attachment point for delay_response…)
⚠ 8 structural-authority     inconclusive (no attachment point for withhold_approvals…)
⚠ 9 state-separation         inconclusive (no database schema discoverable…)

Result
  Level Achieved: —
  Target:         L1 (DURABLE)
  Depth:          V2 (DETERMINISTIC)
  Gate:           FAIL

Every invariant returns hold, violated, or inconclusive. inconclusive is never styled as a pass and never counts toward a level: unmeasured is not passed. That is the whole design.

Install

pip install conformance          # or: uv tool install conformance

Python 3.12+. No account, no API key for verification, nothing uploaded.

Start where you are

conformance init                      # detect the stack, find effect sites, run static checks
conformance doctor --explain-depth    # what each invariant needs to be decidable
conformance verify --path .           # V0 verdicts, no instrumentation required

init needs no SDK, no credentials, and no network. It reports what it found and — for anything it cannot decide at this depth — exactly what would decide it.

Going deeper: record once, verify forever

Declare a scenario naming an entrypoint that runs one bounded episode of your agent, then:

conformance record --scenario myagent --seed 1   # your key, your machine, once
conformance verify --scenario myagent --seed 1   # sealed, offline, byte-deterministic

Your credentials are used exactly once. Everything after that runs against the captured cassette: CI needs no model key and no network. A cassette miss is a loud failure (exit 3) — never a live call, never a guess.

Same commit, same seed, same cassette produces identical verdict bytes and identical witness ids: 1,000 out of 1,000 runs on the crash schedules, Wilson 95% CI ≤ 0.38% flake.

Checking a finding

A violation produces a witness. Export it and it becomes a file anybody can check, offline, without trusting this tool:

conformance export <witness-id>
conformance replay <witness-id>       # re-derives the verdict from the evidence

Editing the evidence breaks its content address and replay refuses.

In CI

- run: pip install conformance
- run: conformance ci --scenario myagent --seed 1 --level-target L1

A violation of an invariant your target level requires fails the check. Exit codes: 0 pass, 1 gate fail, 2 cannot execute, 3 fail-closed breach.

Depth, honestly

Most systems start and stay useful at V0. Moving to V2 means instrumenting the effect boundary — usually a single keyword argument if your effects are HTTP. conformance doctor --scenario <name> --explain-depth lists the exact step per invariant.

Documentation

• Onboarding walkthrough — install to sealed verdict
• Worked examples — manual, HTTP, LangGraph, and CI
• CLI reference
• The specification
• CHANGELOG — including what is knowingly incomplete

Apache-2.0.