record → replay · miss → why · 5-minute demo script

Status: Early alpha — what works and what doesn't

The problem

Coding agents (Claude Code, Codex, etc.) are nondeterministic black boxes. When they break, you can't reproduce it. When they work, you can't prove it'll work again.

                    ┌─────────────────────────┐
  The agent loop:   │  Prompt → LLM → Tools   │──── costs $$$
                    │      ↓          ↓       │──── nondeterministic
                    │  Prompt → LLM → Tools   │──── can't replay
                    │      ↓          ↓       │──── CI needs API keys
                    │  Prompt → LLM → Done    │──── flaky tests
                    └─────────────────────────┘

LLMReplay fixes this by recording the LLM traffic once, then replaying it from disk:

flowchart LR
    A["Agent"] -->|"record"| P["LLMReplay\nProxy"]
    P -->|"forward"| L["Real LLM"]
    P -->|"save"| C["Cassette\n(scrubbed)"]
    
    A2["Agent"] -->|"replay"| P2["LLMReplay\nProxy"]
    P2 -->|"SHA-256\nmatch"| C2["Cassette"]
    P2 -.->|"no network\nneeded"| X["Offline"]

Get started in 30 seconds

One terminal. No CCR. No free keys. No second window.

pip install coding-agent-vcr
llmreplay demo

demo starts a stub gateway + the proxy, records one turn, replays it offline, and prints the commands for a real agent. HMAC is set for you if missing.

Note: CLI / import = llmreplay. PyPI name = coding-agent-vcr.

Real agent (still one terminal)

llmreplay run is the gateway — it starts the proxy, runs your agent, then tears down.

# keep ANTHROPIC_API_KEY in your env (forwarded upstream)
# local HMAC defaults to dev-local-hmac if unset

# Record (proxy = gateway — starts, runs child, tears down)
llmreplay run --mode record --cassette .llmreplay/demo \
  --upstream https://api.anthropic.com \
  -- claude --print "say hi"

# Replay offline
llmreplay run --mode replay --cassette .llmreplay/demo \
  -- claude --print "say hi"

# CI check
llmreplay replay --check --cassette .llmreplay/demo

Miss? → llmreplay why --cassette .llmreplay/demo --request .llmreplay/demo/requests/<tx-id>.json

Full walkthrough: docs/quickstart.md.

How it works

Every field in an LLM request/response is classified into one of four categories:

  ┌─────────────────────────────────────────────────────────────────┐
  │                    Field Classification                         │
  │                                                                 │
  │  ┌──────────┐   Must match. Drives agent behavior.             │
  │  │  STATIC  │   model, messages, tools, tool_choice            │
  │  └──────────┘                                                   │
  │  ┌──────────┐   Noise. Stripped before hashing.                 │
  │  │  IGNORE  │   timestamp, request_id, trace_id                │
  │  └──────────┘                                                   │
  │  ┌──────────┐   Secrets → HMAC placeholders before disk.       │
  │  │   SCRUB  │   API keys, tokens, passwords                    │
  │  └──────────┘                                                   │
  │  ┌──────────┐   Always hit the real endpoint for this step.    │
  │  │   LIVE   │   mark-live Bash, mark-live __llm__              │
  │  └──────────┘                                                   │
  └─────────────────────────────────────────────────────────────────┘

The match pipeline:

flowchart LR
    R["Raw\nRequest"] --> C["Canonicalize\n(JCS)"]
    C --> I["Strip\nIgnored"]
    I --> S["Scrub\nSecrets"]
    S --> T["Sort\nTool Blocks"]
    T --> H["SHA-256\nHash"]
    H --> K["Match\nKey"]

Normative rules: SPEC.md | Architecture: DESIGN.md

Why LLMReplay

Observability shows what happened.
LLMReplay decides what must match and re-executes the trajectory.

Integrations

llmreplay run --mode record -- claude --print "hi"

llmreplay run --mode record -- codex --prompt "hi"

@pytest.mark.llmreplay(cassette=".llmreplay/cassette")
async def test_agent(llmreplay_cassette):
    resp = await llmreplay_cassette.post("/v1/messages", json={...})

Use as a library

from llmreplay import ReplayTransport
import httpx
from pathlib import Path

transport = ReplayTransport(cassette_dir=Path(".llmreplay/cassette"))
async with httpx.AsyncClient(transport=transport, base_url="http://llmreplay") as client:
    resp = await client.post("/v1/messages", json={...})
    assert resp.status_code == 200

Full API: docs/reference/library.md

Hermetic smoke test

git clone https://github.com/dmallya93/llmreplay.git && cd llmreplay
pip install -e ".[dev]"
export LLMREPLAY_HMAC_KEY=dev-local-hmac
./scripts/smoke.sh
# ✓ smoke ok: record→replay (fake upstream)

No Ollama, no paid APIs, no network — pure in-process replay.

Learn

Documentation