Proofline

Proofline is a model-agnostic protocol and reference implementation for verifiable AI runs.

A run bundle is a single JSON document that records the inputs, code revision, model/tool steps, outputs, costs, redactions, and hashes needed to replay, diff, and audit LLM or agent executions. The core is a versioned schema; SDKs, storage backends, and framework adapters are replaceable layers around it.

Why

• Regression testing. proofline diff compares two runs semantically. Run ids, timestamps, and derived digests never show up as noise; what changed in inputs, outputs, and costs does. See CI regression gates for the end-to-end recipe.
• Audit and forensics. Bundles are tamper-evident: a stable SHA-256 digest covers everything that can affect replay decisions, and proofline verify re-checks every stored hash, redaction path, and secret pattern.
• Portability. A bundle is one JSON file with a published schema and spec. No server, no vendor lock-in.

Non-goals

• Not another chat UI or agent framework.
• Not a model provider or prompt registry.
• Not a claim that reruns are bit-identical when the underlying model or tools are nondeterministic.

Install

git clone https://github.com/Powfu-zwx/proofline.git
cd proofline
python -m pip install -e .

Quickstart

Record the same command twice, then prove the runs are semantically identical:

proofline run --out artifacts/a.run.json -- python examples/code_fix_agent.py
proofline run --out artifacts/b.run.json -- python examples/code_fix_agent.py

proofline verify artifacts/a.run.json
# OK artifacts/a.run.json

proofline diff artifacts/a.run.json artifacts/b.run.json
# no semantic differences

Timestamps and run ids differ between the two bundles, but both are excluded from the stable digest and from semantic diffs, so identical work produces an empty diff.

SDK

from proofline import RunRecorder

with RunRecorder(out_path="artifacts/demo.run.json") as recorder:
    with recorder.step("model", "draft", input={"prompt": "hi"}) as step:
        step["output"] = {"text": "ok"}
        step["cost"] = {"input_tokens": 3, "output_tokens": 1}

Secrets are redacted before anything touches disk: keys like api_key / credentials and values like sk-..., AKIA..., Bearer ..., or JWTs are replaced with [REDACTED], and each redaction site is recorded as a JSON Pointer in the bundle.

See examples/rag_citation_check.py and examples/code_fix_agent.py for full agent-shaped runs.

Integrations

OpenAI

python -m pip install -e ".[openai]"

from openai import OpenAI
from proofline import RunRecorder
from proofline.openai import wrap

with RunRecorder(out_path="artifacts/openai.run.json") as recorder:
    client = wrap(OpenAI(), recorder)
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "ping"}],
    )

Each chat.completions.create call is recorded as a model step with the request as input, the response as output, and token usage as cost. Streaming calls are recorded too: the accumulated text is stored together with a truncated flag, and a failed request records an error step. AsyncOpenAI clients are wrapped by the same wrap() call. Run examples/openai_chat.py for an end-to-end recorded call.

Anthropic

python -m pip install -e ".[anthropic]"

from anthropic import Anthropic
from proofline import RunRecorder
from proofline.anthropic import wrap

with RunRecorder(out_path="artifacts/anthropic.run.json") as recorder:
    client = wrap(Anthropic(), recorder)
    client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=256,
        messages=[{"role": "user", "content": "ping"}],
    )

messages.create is recorded the same way, including stream=True event iteration, and AsyncAnthropic clients are wrapped by the same wrap() call. Run examples/anthropic_chat.py for an end-to-end recorded call.

Bundle anatomy

{
  "schema_version": "0.1",
  "run_id": "6f0c0f1e-…",
  "created_at": "2026-08-11T15:00:00.000Z",
  "actor": {"type": "human+agent", "name": "powfu", "version": "0.1.1"},
  "project": {"name": "proofline", "revision": "9dd5f0a…", "dirty": false},
  "invocation": {"argv": ["python", "agent.py"], "cwd": "…", "env_keys": ["PATH"], "python": "3.11.15"},
  "steps": [
    {
      "step_id": "step-1",
      "kind": "model",
      "name": "draft",
      "status": "ok",
      "started_at": "…",
      "ended_at": "…",
      "input": {"prompt": "hi"},
      "output": {"text": "ok"},
      "error": null,
      "cost": {"input_tokens": 3, "output_tokens": 1},
      "metadata": {},
      "input_digest": "…",
      "output_digest": "…"
    }
  ],
  "redactions": [],
  "metadata": {},
  "bundle_digest": "…"
}

Core invariant

A bundle is portable evidence. Any field that cannot affect replay decisions, such as wall-clock timestamps or a fresh run id, is excluded from the stable digest and from semantic diffs. bundle_digest is SHA-256 over canonical JSON of the bundle with volatile fields removed; see the spec for the exact normalization and verification rules.

FAQ

How is this different from LangSmith, Langfuse, or other tracing platforms? Those are observability platforms: hosted dashboards for exploring traces at scale. Proofline is an evidence format: a single verifiable JSON file you can commit to a repo, diff in CI, attach to an incident report, or hand to an auditor. No server, no account, no SDK lock-in. If you already run a tracing platform, proofline is complementary — it is the artifact you keep when a specific run has to be provable.

Is a bundle proof that the model would answer the same way again? No, and the spec is explicit about this non-goal. A bundle proves what was sent, what came back, what it cost, and that nobody altered the record afterwards. Determinism is your pipeline's job; the CI recipe shows how to get there where it matters.

Does redaction make bundles safe to share? Redaction is pattern-based and best-effort — it catches well-known key names and token formats before anything touches disk, and verify re-scans as a second line of defense. It is not a guarantee; review bundles like any fixture before publishing them. See SECURITY.md for the exact boundary.

When should I not use proofline? If you want live dashboards, sampling analytics, or fleet-wide monitoring, use a tracing platform. If your pipeline has no decisions worth auditing or regressing, a bundle is overhead. Proofline earns its keep where runs are consequential: agents that touch code, money, or user data, and pipelines whose behavior changes must be caught in review.

Development

python -m pip install -e ".[dev]"
ruff check src tests examples
pytest -q

License

MIT