agent-assure

Output equivalence is not process equivalence.

Same approval. Missing evidence link. Configured CI gate blocked.

agent-assure catches regressions in declared, structured process evidence that final-answer-only checks can miss. It turns privacy-filtered run records into reproducible comparisons, reviewer-facing artifacts, portable evidence packets, and ordinary CI gate signals. Evidence strength follows its recorded origin: fixture values are authored test inputs, instrumented-adapter values remain producer-attested, and direct model responses do not independently prove that a tool, evidence lookup, policy check, or human review occurred.

Local-first · offline flagship demo · versioned artifacts · CI-native · no hosted control plane required

Run the offline demo · Inspect the reviewer output

For AI leaders · For architects · For engineers

10 deterministic fixture cases · 0 decision fields changed · claim-duration: linked → missing · classification: new_failure · configured CI gate: blocked

Read the flagship demonstration

Choose your path

What it can surface

A final-answer-only check can see no decision-field change while a declared release expectation regresses:

• Evidence and RAG support: a required source or material claim-to-evidence link disappears, or declared corpus and retrieval identity changes. Example: MATERIAL_CLAIM_MISSING_EVIDENCE → new_failure. The development surface also includes an authority-scoped, deterministic controlled evidence-sensitivity contract. Its v1 producer is a declarative synthetic fixture harness, not evidence that a model used contextual evidence instead of parametric memory. A separate repeated paired protocol supports the same narrow binary response relation for prebound stochastic live studies. Its fixed planned-frame composite rate includes every frozen cluster, pessimistically scores a non-analyzable cluster as zero, and reports observed/analyzable counts separately. Verdict-bearing packets and graphs bind the exact candidate RunSet, while both source execution configurations remain anchored to their predeclared arms. Conclusions remain conditional on statistical sufficiency, exact subject binding, and declared cluster assumptions. The experimental research surface also provides a preregistered real-model study workflow, but this repository contains no real-provider study result.
• Human review: a required route or performed-review record is missing.
• Provider, tool, and privacy boundaries: a forbidden provider or tool appears, or declared route, redaction state, or detector identity changes.
• Usage and reliability: retries, tool calls, tokens, latency, rate-limit events, or declared estimated cost change materially.
• Streaming integrity: events are replayed, duplicated, conflicting, or outside the declared sequence contract.
• Protocol-bound live behavior: repeated observations drift outside a declared protocol or comparison boundary, or a paired evidence-response study is underpowered, structurally invalid, or inconsistent with its exact pre-execution arm bindings.

A surfaced difference may be blocking, review-only, or informational. Usage and reliability deltas block only when a suite or policy declares that behavior.

Quickstart

Requires Python 3.11 or newer. The release-qualified matrix is the full suite on Ubuntu 24.04 with Python 3.11 through 3.14, plus the native containment subset on Windows Server 2025 with Python 3.11 and 3.14. macOS is not currently CI-qualified. The five hashed requirements*.lock files are the dependency sets exercised for release; the broader ranges in pyproject.toml express compatibility intent, not exhaustive version-combination qualification.

pip install agent-assure==0.7.0
agent-assure demo flagship --out .tmp/demo/flagship --clean

The installed package runs the bundled deterministic fixture from any directory—no repository clone, provider API key, network call, or token spend.

output equivalence: preserved
missing evidence link: claim-duration
reason code: MATERIAL_CLAIM_MISSING_EVIDENCE
classification: new_failure
CI gate: blocked as expected

The demo wrapper exits 0 only when it verifies that the expected regression was caught. The underlying candidate evaluation, comparison, and CI commands remain strict and exit nonzero for the blocking finding.

Actual reviewer output

_{Screenshot of the reviewer-facing evidence-diff.html produced from the same bundled fixture. Open the image to inspect it at full resolution.}

Reviewer-facing artifacts

Key artifacts are written under .tmp/demo/flagship:

flowchart LR
    subgraph OutputCheck["Ordinary visible-output check"]
        BOut["Baseline output<br/>recommendation=approve<br/>outcome=approve"]
        COut["Candidate output<br/>recommendation=approve<br/>outcome=approve"]
        Same["Visible answer unchanged"]
        BOut --> Same
        COut --> Same
    end

    subgraph InvariantCheck["agent-assure invariant check"]
        BEv["Baseline evidence<br/>claim-duration linked"]
        CEv["Candidate evidence<br/>claim-duration missing link"]
        Pass["Baseline evaluation: pass"]
        Fail["Candidate evaluation: fail<br/>MATERIAL_CLAIM_MISSING_EVIDENCE"]
        BEv --> Pass
        CEv --> Fail
    end

    Same --> Tension["Output unchanged<br/>but governance invariant regressed"]
    Equiv["Fixture equivalence: pass"] --> Compare["Baseline-to-candidate comparison"]
    Pass --> Compare
    Fail --> Compare
    Tension --> Compare

    Compare --> NewFailure["Classification: new_failure"]

Where it fits

agent-assure complements the evaluation, observability, runtime-control, and governance systems teams already use.

It is a particularly strong fit when release review must be local, reproducible, CI-enforceable, and traceable without a required hosted control plane.

How it works

Declare → Capture from a declared source (privacy-filtered) → Evaluate
        → Compare (when equivalent) → Packet → Gate

Declared expectations and canonical run evidence remain distinct. The candidate is evaluated first; equivalent-baseline context is added only after comparison prerequisites pass. The evidence packet then supports a CI signal and human release review.

agent-assure is a local-first Agent Release Assurance Compiler: controls and evidence stay bound to method identity, prerequisites, provenance, assumptions, and limitations.

The assurance model is deliberately bounded:

• Deterministic and reproducible in fixture mode: fixed, versioned fixtures, canonical serialization, schemas, and digest-bound manifests make checks repeatable; that reproducibility does not estimate production prevalence.
• Scoped invariance claims: results cover only declared, observable fields and explicit prerequisites—not hidden reasoning or all production behavior.
• Traceable lineage: expectations connect to RunSets, findings, comparisons, evidence packets, and configured gate state; provenance records participating material.
• Fail-closed: malformed, conflicting, incompatible, ambiguous, or unbound evidence does not silently become a passing review.
• Statistically bounded: live conclusions about probabilistic provider behavior remain tied to a declared statistical protocol, with its data boundary, configuration, window, sampling noise, dependence, and limitations explicit.

Architecture choices and evidence boundaries are documented in architectural decision records (ADRs), including deterministic fixture versus stochastic live semantics.

Challenge the assurance controls

The development-RFC core/v1 mutation catalog runs seven deterministic challenges across evidence linkage, human-review routing, tool boundaries, provenance identity, privacy redaction, duplicate replay, and budget-stop integrity:

agent-assure controls mutate \
  --suite assurance/suite.yaml \
  --runset runs/baseline.json \
  --catalog core/v1 \
  --seed 0 \
  --today 2026-08-02 \
  --full-report \
  --out reports/control-challenge

Every selected operator runs independently against the same immutable source. The output binds the canonical catalog digest, normative expected detector, observed and prohibited substitute findings, exact changed paths, provenance, independence class, seed, and limitations. The campaign itself remains a finite challenge record. controls efficacy can derive an exact catalog-relative detector kill ratio over completed applicable outcomes while preserving inapplicable, invalid, and error counts outside its denominator. That ratio is not a safety score, statistical confidence interval, or universal-coverage claim.

Only deterministic caught and survived campaign outcomes may contribute a control-efficacy verdict. An applicable critical threat with no completed challenge emits CRITICAL_THREAT_UNCOVERED and requires review under the default profile. Required and critical survivors, invalid/error outcomes, and required non-verdict outcomes are always blocking. Evidence-packet ci gate requires control-efficacy evidence by default and uses strict verification when it is present; pass a verifier-owned controls-mutation YAML with --efficacy-policy. --require-efficacy remains as an explicit restatement for existing automation. Strict CI accepts only complete, all-caught, all-applicable-challenged evidence and pins the catalog, selected and required operators, and threat manifest. The conspicuous --allow-missing-efficacy-for-migration escape hatch is only for temporary non-assurance migration and records that weaker profile. Use --allow-advisory-efficacy only for an explicit review flow with evidence present.

For an efficacy-bearing release claim, ci gate --release-profile additionally requires an evidence packet, a verifier-owned --efficacy-policy, present efficacy evidence, and blocking warning/not-evaluated handling. It rejects the advisory and compatibility weakening flags. This is a CI efficacy profile, not publication authorization. For v0.7.0, make release-publish-check checks the committed bounded claim profile, replays the separately staged seven-operator synthetic fixture packet under its verifier policy, and then runs the complete engineering release checks. make empirical-readiness remains a separate claim-qualification target and cannot add empirical claims when its evidence is absent. This validates the committed packet's internal bindings and policy projection; it does not rerun the operators or attest fresh campaign execution.

The v0.7.0 release line is bounded to engineering qualification and committed deterministic-fixture artifact validation. No real-model study, qualifying external pilot, independent empirical review, or frozen confirmatory benchmark is included. The release therefore makes no empirical-effectiveness, external-validity, population-generalization, production-control-effectiveness, provider-quality, safety, compliance, or deployment-fitness claim. Those artifacts are prerequisites only for the corresponding empirical claim, not for distribution of this bounded release.

The packaged offline demonstration exercises both a strong and deliberately weakened assurance control, then verifies that an unrelated failure cannot substitute for the expected detector:

agent-assure demo assure-the-assurance \
  --out .tmp/demo/assure-the-assurance \
  --clean

For a minimal editable workflow, start with agent-assure init controls-mutation, run the read-only doctor controls-mutate preflight, then produce a campaign and control-efficacy-report.json.

Inspect the exact seven-operator catalog · Measure control efficacy · Read "Who assures the assurance?" · Review the evidence contracts

Integrate your agent

agent-assure integrates through declared YAML expectations, versioned run evidence, the documented CLI, and the framework-neutral AgentRunRecord producer contract.

The integration contract has three parts:

1. Declare structured process-evidence expectations in YAML.
2. Produce versioned run records from fixtures or privacy-filtered adapters with explicit field origins.
3. Evaluate the candidate, compare equivalent baseline evidence when available, and gate the resulting evidence packet.

A real expectation from the bundled flagship suite:

cases:
  - case_id: shared-source-multi-claim
    fixture_id: shared-source-multi-claim
    expectation:
      expected_recommendation: approve
      required_evidence_refs:
        - ref-shared-clinical-note
      material_claim_ids:
        - claim-eligibility
        - claim-duration

agent-assure does not infer material claims from rationale text. Authors declare the oracle, and run-record producers emit explicit claim-to-evidence links for the material claims they intend to satisfy.

Author expectations · Understand the CLI contract · Review the public API surface · Understand evidence packets

GitHub Actions example using the bundled fixture

Pin the runner image, GitHub-owned actions, package, and composite action in release workflows. This runnable example targets v0.7.0. Replace the example suite and variant paths with your own controlled materials.

name: agent-assure
on: [pull_request]

permissions:
  contents: read

jobs:
  assure:
    runs-on: ubuntu-24.04
    steps:
      # actions/checkout@v7.0.1
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      # actions/setup-python@v7.0.0
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97
        with:
          python-version: "3.11"
      - run: python -m pip install agent-assure==0.7.0
      # agent-assure v0.7.0
      - uses: acblabs/agent-assure/.github/actions/agent-assure@45d5855d31e30e616fa497aaf07ccc51472034f0
        with:
          suite: examples/prior_auth_synthetic/suite.yaml
          baseline-variant: examples/prior_auth_synthetic/variants/baseline.yaml
          candidate-variant: examples/prior_auth_synthetic/variants/candidate_evidence_normalization.yaml
          report-mode: full
          upload-reports: "true"

full produces the complete review artifacts; fail-fast gives shorter blocking feedback. The bundled fixture invocation is explicitly a non-assurance smoke example, not evidence of control efficacy. The v0.7.0 composite action has two mutually exclusive paths. Its strict path requires both control-efficacy-report and efficacy-policy, rebuilds the current evaluation packet with that report, and strictly re-verifies it against the separately trusted policy. Repository CI qualifies that strict path on a hosted Ubuntu runner with a deterministic passing efficacy campaign and repeats the strict packet gate; a separate runner job retains the explicitly labeled non-assurance migration smoke. This qualifies action behavior, not the independence of production evidence. Keep the production policy under verifier control (for example, in a separately protected checkout placed beneath GITHUB_WORKSPACE); a policy supplied by the candidate is not independent authority. Fixture-only, evaluation-only jobs must instead opt into allow-missing-efficacy-for-migration: "true", which remains a labeled non-assurance result. Omitting both paths, supplying only one strict input, or combining strict inputs with the migration flag fails closed. Baseline-bearing runs remain blocking for both new_failure and persistent_failure dispositions; an exact waiver does not authorize either comparison result. The v0.7.0 action accepts out-dir only as a strict descendant of GITHUB_WORKSPACE or RUNNER_TEMP and rejects traversal, roots, and linked/reparse-point ancestors. Strict efficacy mode further requires the output beneath GITHUB_WORKSPACE, matching the packet verification root. In the v0.7.0 action, reports remain local to the runner by default. Set upload-reports: "true" only after approving GitHub artifact retention; that opt-in uploads the packet, its privacy-filtered assurance evidence graph, manifest, summaries, and CI diagnostics. Set upload-full-artifacts: "true" only when the workflow is also approved to retain compiled suites, fixture data, and RunSets. The default retention period for either explicit upload is 14 days.

Runtime coordination locks

Atomic publishers intentionally leave hidden coordination files beside their output targets so another process cannot exploit lock unlink/replacement races. If that parent directory is inside your repository, add this narrow rule to your repository's .gitignore (the package cannot update it for you):

.agent-assure-*.lock

These lock files contain no assurance evidence or credentials and normally remain on disk for reuse. Do not replace the rule with *.lock, which would also hide dependency lockfiles and unrelated project state.

Integrations and maturity

Release line: v0.7.0 (Beta). The package and primary CLI contracts are qualified under the bounded engineering profile described above. Publication of the package is not publication of a real-model study, external pilot, or production-effectiveness result.

The CLI, YAML authoring format, persisted versioned JSON artifacts, and AgentRunRecord producer contract are the primary integration surface. Framework adapters, streaming, and live execution remain experimental. The Beta label applies only to the primary surface; development-RFC contracts remain non-stable. PyPI's Development Status :: 4 - Beta is the closest standardized classifier and does not widen that surface.

Want to help with optional empirical-claim research? The short external pilot quickstart targets 10–15 minutes of participant effort in a non-maintainer-controlled fork; CI runtime may be longer. It needs no participant-supplied model key, proprietary data, or repository secret. Interest or a workflow run is not evidence completion: capture, participant friction/finalization and prospective consent, and a byte-bound human review are all required for the pilot bundle.

Framework adapters project only privacy-filtered agent_assure metadata into the framework-neutral run-record model. They ignore raw prompts, messages, completions, tool arguments, token chunks, and unredacted summaries.

Governance crosswalks

Packet-resident evidence can be mapped to selected concepts in the NIST AI RMF, OWASP Top 10 for LLM Applications 2025, ISO/IEC 42001, and MITRE ATLAS 2026.06.

These crosswalks are planning and review aids. They do not establish framework conformance, complete coverage, third-party assurance, or endorsement.

Claim boundary

Measured evidence, not a blanket trust claim.

This project is not a compliance attestation.

Generated artifacts make declared inputs, findings, limitations, and gate state traceable and auditable for human review. Whether the release decision is defensible remains a human and organizational judgment. agent-assure does not determine safety or replace domain, legal, regulatory, clinical, security, provider-quality, model-quality, or business-impact review.

Pattern redaction is a guardrail, not comprehensive DLP or de-identification. Live conclusions remain bounded by the declared protocol, data boundary, provider/model configuration, and execution window. Review the claim boundary, limitations, threat model, privacy model, and security guidance.

Learn more

• Statistical methods: Repeated paired evidence sensitivity · Preregistered real-model study · Live calibration

• Start: Documentation · For AI leaders · For architects · For engineers

• Demos: Assure the assurance · Flagship · RAG provenance · Expense approval

• Integrations: LangGraph · Google ADK · Adapter contract

• Assurance: What this measures · Control efficacy · Evidence packets · Live calibration

• Evidence-carrying releases: Core mutation catalog · Minimal evidence graph · Contracts and campaign guide · Architecture · CLI contract

• Security and governance: Claim boundary · Threat model · Security correction containment · Governance crosswalks

• Project: Contributing · Changelog · License

pip install -e ".[dev]"
git config core.hooksPath .githooks
python scripts/check_docs_alignment.py
ruff check .
mypy src scripts
pytest
python -m build

Citing

This project ships a CITATION.cff. Use GitHub’s Cite this repository control for generated citation formats.