assurance-cli

Did the job cover everything it was supposed to cover?

One command. One honest ratio. An exit code your pipeline can act on.

pip install assurance-cli

assurance diff --expected corpus.txt --found retrieved.json \
  --scope "documents the question spans" --where "the retrieved set" --fail-on-gap

2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
also present and not expected: doc-9

That second line matters as much as the ratio: the retriever drew on something the scope never allowed. It's reported, and it earns no credit.

No account · no API key · no network call · no model decides any of it

Three commands

diff is the general one. check is the special case for a folder of dated or numbered tabular files — if your files are .md, or named in a format it can't read, use diff and declare the set yourself.

assurance pin — did an MCP server change what it tells the model?

CVE-2025-54136 (CVSS 8.8): approving a tool definition does not survive subsequent server-side changes. The server you approved in March can serve a different description in August — same client, same name, no re-prompt.

pip install 'assurance-cli[mcp]'

assurance pin --save                 # snapshot every tool your MCP servers expose
assurance pin --check                # exit 1 if any definition changed since the snapshot

Pins live in .assurance/mcp-pins.json — commit it like a lockfile and review changes in PRs. Stdio servers only in this release; HTTP/SSE transports are named and skipped.

CI gate (no account, no service, no model):

- run: pip install 'assurance-cli[mcp]'
- run: assurance pin --check

Exit 1 means a definition moved and needs a human look. Exit 2 means the gate could not run (missing mcp extra, no config, no pin file yet).

assurance drift — is the failure rate shifting?

Control chart over any binary outcome stream. Needs at least 21 runs (20 baseline + 1 monitored). Refuses below that with a message naming how many more are needed. No model, no labels.

assurance drift events.jsonl --field outcome --failure verification_failed
assurance drift results.csv  --column status --failure error --baseline 0.05

Exit 1 when a shift is detected — a CI gate the same way pin --check works.

- run: assurance drift outcomes.jsonl --field outcome --failure error

assurance diff — any two sets of keys

# code review agent actually read the diff?
git diff --name-only origin/main...HEAD > changed.txt
assurance diff --expected changed.txt --found reviewed.txt --fail-on-gap

# eval suite ran every declared case?
assurance diff --expected cases.json --found ran.json --fail-on-gap

# straight from a pipe
retriever --query "$Q" | jq -r '.chunks[].doc_id' | \
  assurance diff --expected corpus.txt --found - --json

Inputs are whatever you already have: one key per line, a JSON array (strings, or objects with key/id/name/path), - for stdin, or an inline comma list.

assurance check — a folder of dated or numbered files

This one is for CI, not for an agent. Anything with a shell will list the directory and spot the gap itself. We tested that and the run without our tool did better. The value here is a gate with no model in it, returning the same exit code every time.

assurance check ~/reports

22 of 24 months from 2024-01 to 2025-12 in reports — not in this folder: March 2025, July 2025
— Range inferred from filenames: earliest 2024-01, latest 2025-12. Override with --from / --to.

assurance check ~/invoices --expect numbered

7 of 8 runs from inv_0001 to inv_0008 in invoices — not in this folder: INV-0006
— Range inferred from filenames: earliest inv_0001, latest inv_0008. Override with --from / --to.

Monthly, quarterly, weekly, daily, numbered. That last line is the derivation: it prints with every ratio so you can disagree with the denominator, not just the result.

When a file is there under a name it can't read, it says so beside the gap, because that's the difference between never produced and produced and named differently:

11 of 12 months from 2025-01 to 2025-12 — not in this folder: March 2025
— 1 name here could not be read as any of them: March FINAL v2.csv

assurance init — did anything change underneath?

assurance init ~/thesis-data
# Baseline written to ~/thesis-data/.assurance.json — 34 tabular files recorded.

# ... weeks pass, several people touch the folder ...
assurance check ~/thesis-data --against-baseline

Exit codes

"I couldn't check this" exits 1, not 0. A folder whose filenames it can't parse must not look like a folder it checked and found whole.

Diagnostics go to stderr, results to stdout, so --json stays pipeable.

It expects your files, not tidy ones

• Excel exports work. UTF-8 BOM and CRLF are handled; a BOM used to glue itself to your first key and report it as missing and unexpected in the same sentence
• Spaces, unicode and month words in filenames — Inventory Report August 2024.csv parses
• .xlsx, and nested subfolders
• A piped CSV is refused, not misread. It names the column-picking command instead of quietly admitting your header row as a key
• When it can't work out a series it says so, rather than reporting an empty check as a pass

Use it for

What it won't do

• Invent your expected set. diff takes your declaration; check derives one and prints how
• Send anything anywhere. No network, no telemetry, no keys
• Guess. A JSON object of id → metadata is refused, not interpreted

Family

assurance-core — the pure arithmetic, zero dependencies · assurance-mcp — the same checks as MCP tools

Upstream is I-Ops; this repo is a publication, never a source. Apache-2.0.