Proofline

Proofline is a model-agnostic protocol and reference implementation for verifiable AI runs.

A run bundle is a single JSON document that records the inputs, code revision, model/tool steps, outputs, costs, redactions, and hashes needed to replay, diff, and audit LLM or agent executions. The core is a versioned schema; SDKs, storage backends, and framework adapters are replaceable layers around it.

Why

• Regression testing. proofline diff compares two runs semantically. Run ids, timestamps, and derived digests never show up as noise; what changed in inputs, outputs, and costs does. See CI regression gates for the end-to-end recipe.
• Replay. A bundle doubles as a test fixture: recorded responses are served back through the wrappers, so pipelines re-run deterministically, offline, and for free — and diffing a replayed run against its baseline tells you whether a change came from your code or from model drift. See replay.
• Audit and forensics. Bundles are tamper-evident: a stable SHA-256 digest covers everything that can affect replay decisions, and proofline verify re-checks every stored hash, redaction path, and secret pattern.
• Portability. A bundle is one JSON file with a published schema and spec. No server, no vendor lock-in.

Non-goals

• Not another chat UI or agent framework.
• Not a model provider or prompt registry.
• Not a claim that reruns are bit-identical when the underlying model or tools are nondeterministic.

Install

pip install proofline

Provider integrations are optional extras: pip install "proofline[openai]" or pip install "proofline[anthropic]". For development installs, see CONTRIBUTING.md.

Quickstart

Record the same command twice, then prove the runs are semantically identical:

proofline run --out a.run.json -- python -c "print('hello agent')"
proofline run --out b.run.json -- python -c "print('hello agent')"

proofline verify a.run.json
# OK a.run.json

proofline diff a.run.json b.run.json
# no semantic differences

Timestamps and run ids differ between the two bundles, but both are excluded from the stable digest and from semantic diffs, so identical work produces an empty diff. For agent-shaped runs with tool, model, and check steps, clone the repo and try examples/code_fix_agent.py and examples/rag_citation_check.py.

SDK

from proofline import RunRecorder

with RunRecorder(out_path="artifacts/demo.run.json") as recorder:
    with recorder.step("model", "draft", input={"prompt": "hi"}) as step:
        step["output"] = {"text": "ok"}
        step["cost"] = {"input_tokens": 3, "output_tokens": 1}

Secrets are redacted before anything touches disk: keys like api_key / credentials and values like sk-..., AKIA..., Bearer ..., or JWTs are replaced with [REDACTED], and each redaction site is recorded as a JSON Pointer in the bundle.

See examples/rag_citation_check.py and examples/code_fix_agent.py for full agent-shaped runs.

Replay

Any bundle recorded through the wrappers can answer the same pipeline again — no API key, no network, no cost:

from proofline.replay import ReplaySource

client = wrap(OpenAI(), recorder, replay=ReplaySource("baseline.run.json"))

Or, with zero code changes, set PROOFLINE_REPLAY=baseline.run.json in the environment. Strict matching turns the baseline into a fixture; ordered matching lets a changed pipeline complete so the bundle diff shows exactly what your code changed. docs/replay.md covers strategies, streaming fidelity, and the attribution workflow.

Integrations

OpenAI

python -m pip install -e ".[openai]"

from openai import OpenAI
from proofline import RunRecorder
from proofline.openai import wrap

with RunRecorder(out_path="artifacts/openai.run.json") as recorder:
    client = wrap(OpenAI(), recorder)
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "ping"}],
    )

Each chat.completions.create call is recorded as a model step with the request as input, the response as output, and token usage as cost. Streaming calls are recorded too: the accumulated text is stored together with a truncated flag, and a failed request records an error step. AsyncOpenAI clients are wrapped by the same wrap() call. Run examples/openai_chat.py for an end-to-end recorded call.

Anthropic

python -m pip install -e ".[anthropic]"

from anthropic import Anthropic
from proofline import RunRecorder
from proofline.anthropic import wrap

with RunRecorder(out_path="artifacts/anthropic.run.json") as recorder:
    client = wrap(Anthropic(), recorder)
    client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=256,
        messages=[{"role": "user", "content": "ping"}],
    )

messages.create is recorded the same way, including stream=True event iteration, and AsyncAnthropic clients are wrapped by the same wrap() call. Run examples/anthropic_chat.py for an end-to-end recorded call.

Bundle anatomy

{
  "schema_version": "0.1",
  "run_id": "6f0c0f1e-…",
  "created_at": "2026-08-11T15:00:00.000Z",
  "actor": {"type": "human+agent", "name": "powfu", "version": "0.2.0"},
  "project": {"name": "proofline", "revision": "9dd5f0a…", "dirty": false},
  "invocation": {"argv": ["python", "agent.py"], "cwd": "…", "env_keys": ["PATH"], "python": "3.11.15"},
  "steps": [
    {
      "step_id": "step-1",
      "kind": "model",
      "name": "draft",
      "status": "ok",
      "started_at": "…",
      "ended_at": "…",
      "input": {"prompt": "hi"},
      "output": {"text": "ok"},
      "error": null,
      "cost": {"input_tokens": 3, "output_tokens": 1},
      "metadata": {},
      "input_digest": "…",
      "output_digest": "…"
    }
  ],
  "redactions": [],
  "metadata": {},
  "bundle_digest": "…"
}

Core invariant

A bundle is portable evidence. Any field that cannot affect replay decisions, such as wall-clock timestamps or a fresh run id, is excluded from the stable digest and from semantic diffs. bundle_digest is SHA-256 over canonical JSON of the bundle with volatile fields removed; see the spec for the exact normalization and verification rules.

FAQ

How is this different from LangSmith, Langfuse, or other tracing platforms? Those are observability platforms: hosted dashboards for exploring traces at scale. Proofline is an evidence format: a single verifiable JSON file you can commit to a repo, diff in CI, attach to an incident report, or hand to an auditor. No server, no account, no SDK lock-in. If you already run a tracing platform, proofline is complementary — it is the artifact you keep when a specific run has to be provable.

Is a bundle proof that the model would answer the same way again? No, and the spec is explicit about this non-goal. A bundle proves what was sent, what came back, what it cost, and that nobody altered the record afterwards. Determinism is your pipeline's job; the CI recipe shows how to get there where it matters.

Does redaction make bundles safe to share? Redaction is pattern-based and best-effort — it catches well-known key names and token formats before anything touches disk, and verify re-scans as a second line of defense. It is not a guarantee; review bundles like any fixture before publishing them. See SECURITY.md for the exact boundary.

When should I not use proofline? If you want live dashboards, sampling analytics, or fleet-wide monitoring, use a tracing platform. If your pipeline has no decisions worth auditing or regressing, a bundle is overhead. Proofline earns its keep where runs are consequential: agents that touch code, money, or user data, and pipelines whose behavior changes must be caught in review.

Development

python -m pip install -e ".[dev]"
ruff check src tests examples
pytest -q

License

MIT