Decision Contract Audit

An Inspect AI extension that audits an LLM (or any) decision point against six reliability contracts by replaying a frozen decision log. No model calls, no API keys: every run goes through Inspect's mockllm and exits with a machine-checkable report.

The six contract families:

Scope is deliberate: this is post-hoc certification of caller-selected cases, not runtime prevention. A residual is a supplied case outside every cage rule; it is not automatically unsafe unless the caller selected it on that basis.

Install

pip install decision-contract-audit

Quickstart

The runnable example lives in the repository, not the distribution:

git clone https://github.com/CTWalk/decision-contract-audit
cd decision-contract-audit
pip install .
python examples/custom_decision_point.py

The example replays a synthetic loan-approval triage log with planted violations in five of the six families and exits nonzero unless every one is detected — it doubles as the repository's keyless self-check (the same command CI runs).

Wire your own decision point

A decision point is: an ordered set of options (least → most hesitant), a set of states, decisions for those states (a replay log, a dict, or a callable), and the rules you want enforced.

from decision_contracts import run_contracts

report = run_contracts(
    decision_order=["approve", "review", "decline"],  # least → most hesitant
    states={"low": {"score": 1}, "high": {"score": 3}},
    decisions={"low": "review", "high": "approve"},   # or a log, or a callable
    invariant_rules=[{
        "name": "score floor",
        "predicate": {"field": "score", "op": "<", "value": 2},
        "forbidden": ["approve"],
    }],
    monotonic_pairs=[("low", "high")],                # optional
    cage_cases=[{"state_id": "high", "decision": "approve"}],
    cage_rules=[{                                     # optional
        "name": "hard floor",
        "predicate": {"field": "score", "op": "<", "value": 1},
        "forbidden": ["approve"],
        "fallback": "review",
    }],
)

print(report["passed"], "—", report["reason"])
for family, row in report["contracts"].items():
    print(f"{family:15} {'PASS' if row['passed'] else 'FAIL'} — {row['reason']}")

Expected output: every configured family passes except cage, which fails with 1 valid decision(s) were residual to all cage rules — the caller selected the high approval for certification, but the toy rule (score < 1) does not cover it. Decide whether that gap is acceptable or tighten the rule set.

report["contracts"]["cage"] carries the certification counts first-class: covered_count, overridden_count, and residual_count. Supplying cage_rules requires an explicit cage_cases set, so ordinary benign decisions are never silently relabeled as residuals.

Malformed inputs fail closed: an unknown state, an unresolvable pair, or an unsupported rule operator produces a failing score with a reason, never a silent pass.

For finer control (running inside your own Inspect pipeline), use build_contract_task(...), which returns a plain Inspect Task.

Provenance

• Built and verified against inspect-ai 0.3.248 (2026-07-20); requires Python ≥ 3.10, matching Inspect AI's floor (CI exercises 3.10 and 3.12). The scorers replay pre-recorded data through a no-generation solver, and the example fails unless Inspect reports zero model usage.
• The contract semantics were transferred from a private research harness in which they reproduced 12/12 published experimental records and 9/9 adversarial edge cases against a frozen expectations corpus, and the cage certification reproduced its reference log's covered/residual split exactly (audit executed 2026-07-20). Those frozen artifacts contain full experimental history and remain in the private provenance archive; this repository ships only the extension and a synthetic worked example.
• Development was AI-assisted under human review — see NOTICE.

License

Apache-2.0 — see LICENSE and NOTICE.