evidence-readiness

Checks whether an agent execution trace contains the evidence a post-hoc causal reconstruction would require.

When an AI agent causes a loss — a bad config change, a cross-session data leak, contaminated output cascading between agents, a prompt injection — someone eventually has to reconstruct why from the trace. Traces from general-purpose agent instrumentation cannot support that reconstruction: the evidence was never recorded. This repository contains the evidence-readiness specification (SPEC.md), a checker that evaluates a trace against it, and reference fixtures showing what conformant instrumentation looks like.

What this is — and is not

• It checks that evidence is present, not that it is true. A fabricated-but-internally-consistent trace passes. This tool authenticates nothing, and its result must never be relayed as "this agent is safe."
• It never determines whether a failure occurred. It answers "does this trace contain what a reconstruction of class X would require," never "did class X occur." It contains no detection or scoring logic.
• This repository does not contain the reconstruction engine. The specification is published so the method is assessable; the engine is available for inspection under the terms in SOURCE_AVAILABILITY.md.
• Four failure classes are covered: config_drift, session_isolation_failure, cascade_contamination, prompt_injection. A fifth (tool_misuse) was investigated and declined for stated structural reasons — a documented negative result, not a gap (SPEC.md §8).
• The spec is a profile of the OTel GenAI semantic conventions: core OTel plus documented extensions (SPEC.md §2). It carries no standards-body endorsement.

Who it is for

• Engineers instrumenting agents who want incidents to be reconstructable after the fact — diff your trace against a conformant fixture and see what to add.
• Reviewers (insurers, auditors, counterparties) evaluating whether an agent system's telemetry would support retrospective loss adjudication.
• Coding agents: see llms.txt, spec.json (the machine-readable spec export, generated — never hand-maintained), and the MCP server under mcp/ (configuration below).

Quickstart (60 seconds)

# from a clone of this repository
pip install .
evr check path/to/trace.json        # or a directory of traces
evr check --json path/to/traces/    # machine-readable report
evr check --require-class prompt_injection path/to/trace.json

The package has zero runtime dependencies; pip install . in any Python ≥3.10 environment is the whole setup. Equivalent uvx forms (uvx --from . evr check ./traces from a clone; uvx evidence-readiness check ./traces once released) are documented here but have not been exercised in this build.

Exit codes carry no reconstruction vocabulary: 0 all requirements met · 1 one or more requirements partially met · 2 one or more requirements unmet · 3 malformed input (SPEC.md §9 is the source of record). Universal requirements (EVR-R1..R6) drive the exit code; per-class evidence readiness is reported separately, and --require-class folds a named class's readiness into the exit code. For every failed requirement the output names the specific missing field and the span kind it belongs on.

Every run ends with the same statement this README opened with: presence, not truth — evidence being present does not make it authentic, and a passing trace is not a safe agent.

CI: check every PR (GitHub Action)

A spec is read once; a check that runs on every PR changes instrumentation behavior. Add this to a workflow in the repository that produces your traces (evidence-readiness is this repository's owner once published; from within this repository itself, uses: ./.github/actions/readiness-check):

steps:
  - uses: actions/checkout@v4
  - uses: evidence-readiness/spec/.github/actions/readiness-check@main
    with:
      trace-path: traces/

That is the whole setup — the default GitHub runner's python3 is sufficient, and the Action installs the (zero-dependency) checker itself. The step fails when the checker's exit code exceeds minimum-passing-exit-level (default 0; exit-code semantics as in the Quickstart above). Optional required-classes folds named classes' evidence requirements into the exit code. Outputs: exit-code, exit-meaning, report-path (full JSON report), and per-class-readiness. PR annotations name each incomplete requirement's specific missing field and span kind. Every annotation is a statement about the trace telemetry, never about the agent: an annotated PR has incomplete evidence, not a detected failure — and a clean run is not a safe agent. This repository's own CI (.github/workflows/ci.yml) dogfoods the Action against fixtures/, asserting the expected — deliberately nonzero — exit codes per bucket.

MCP server (agent surface)

mcp/server.py exposes the checker and the requirement data to agents over the Model Context Protocol (stdio). Three tools: check_trace_readiness (file, directory, or inline JSON — inline content is processed in memory and never written anywhere), get_requirement, and list_class_requirements. Configuration:

{
  "mcpServers": {
    "evidence-readiness": {
      "command": "python3",
      "args": ["/absolute/path/to/evidence-readiness/mcp/server.py"]
    }
  }
}

It runs from a clone without installation and needs no dependencies. It is local-only: no network calls, no phone-home, nothing persisted. It deliberately has no submission tool — the server never transmits anything; submitting a trace anywhere is a human decision gated on the CONTRIBUTING.md disclaimer and a redaction judgment an agent cannot make. Every tool description carries the presence-not-truth statement, so a readiness result cannot reasonably be relayed as "this agent is safe."

Status

The specification is versioned (SPEC.md carries its current version and correction log) and the checker is implemented against it, with reference fixtures (fixtures/), the machine-readable export (spec.json), the agent surface (llms.txt, mcp/), and the CI action (.github/actions/readiness-check/) in place. The methodology and limitations paper is drafted (PAPER.md); its front-matter banner records which sections have had an adversarial read and which have not. The standing disclosure policy is SOURCE_AVAILABILITY.md; reporting and trace-submission terms are CONTRIBUTING.md, with redaction guidance in REDACTION.md and release verification in RELEASING.md. Corrections are logged in CHANGELOG.md. The spec is descriptive: every requirement is backed by an observed outcome in a validation record of 240 reconstruction attempts, 222 of them synthetic — backing is per-requirement, from the specific cases that exercised it, not a property measured across all 240 (18 real-world Who&When cases validate the abstention direction only; the split is stated wherever the count appears — SPEC.md §1).

Attribution

Brien Buckman

Licensed under Apache-2.0 (see LICENSE).

The substance of this repository's development record is public: corrections that affect published content are logged in CHANGELOG.md, and the development defect record, as it bears on the method, is summarized in PAPER.md §8.6. The commit history in which these files were drafted and revised is not published.