llm-run-recorder 0.2.2

Flight recorder for LLM agents: trace every call, replay runs deterministically, diff runs to find where behavior diverged.

AgentRewind

Flight recorder for LLM agents. Trace every LLM and tool call an agent makes, replay a run deterministically (no API calls, no cost, no nondeterminism), and diff two runs to find exactly where their behavior diverged.

"It worked yesterday — why is it different today?" is the defining debugging problem of agent development. Tracing tools show you what happened; AgentRewind also lets you re-execute what happened and compare runs structurally.

Install

pip install llm-run-recorder            # core (stdlib-only)
pip install 'llm-run-recorder[server]'  # + web trace viewer

Trace

import agentrewind as al

@al.traced(kind="tool")
def search(query: str) -> str:
    ...

with al.trace("research-agent"):
    plan = call_llm(...)          # record with al.record_llm_call(...) or a Recorder
    evidence = search(plan.query)

Traces (span tree, inputs/outputs, latency, token counts, errors) are stored in a local SQLite db at ~/.agentrewind/traces.db — no account, no server required.

Sensitive-data controls

For production traffic, opt into recursive persistence-time redaction. It removes common credential fields (api_key, authorization, password, secret, and token) and common token formats from trace metadata, span payloads, and the local replay cache:

import agentrewind as al

al.configure(redaction=al.RedactionPolicy())

Redaction intentionally also applies to cached responses, so use it where protecting local data matters more than replaying the original secret verbatim.

Replay

Wrap your provider call in a Recorder. Responses are cached by a canonical request fingerprint; in replay mode the whole agent run re-executes deterministically and offline:

from agentrewind.replay import Recorder

llm = Recorder(call_openai, mode="record")   # live run, responses cached
llm = Recorder(call_openai, mode="replay")   # deterministic re-run, zero API calls
llm = Recorder(call_openai, mode="auto")     # replay on hit, record on miss

Or instrument an SDK client in place — existing code keeps calling it exactly as before:

import agentrewind as al

client = al.instrument(OpenAI(), mode="auto")       # or Anthropic()
client.chat.completions.create(model="gpt-4o", messages=[...])  # traced + replayable

(Prefer explicit wrappers? agentrewind.providers.OpenAIChat / AnthropicMessages do the same without monkey-patching.)

Streaming is captured too: Recorder.call_stream / acall_stream pass chunks through live while recording them, and replay re-streams the identical chunks offline.

Requests containing volatile fields (timestamps, request ids) can be normalized before fingerprinting with Recorder(..., canonicalize=strip_volatile) so they still replay.

Async agents are supported throughout: @al.traced works on async def functions (context propagates through awaits and asyncio.gather), and Recorder.acall is the awaitable variant for async provider clients.

Diff

$ agentrewind diff 3f2a91 8c17d0
2 divergence(s); first divergence is where the runs split:

1. [input] /llm:mock-4o — inputs differ
     left : {"messages":[{"content":"decide next step",...
     right: {"messages":[{"content":"decide the next step",...
2. [output] /tool:search — outputs differ

The diff walks both span trees in parallel and reports structure, input, and output divergences in execution order — entry #1 is where the runs first split.

CLI & viewer

agentrewind list                 # recent runs
agentrewind show <trace-id>      # span tree with latencies
agentrewind diff <run1> <run2>   # structural diff (exit code 2 if runs differ)
agentrewind serve                # web viewer at http://127.0.0.1:4317
agentrewind export <trace-id> -o baseline.json  # portable artifact for a bug report or CI fixture
agentrewind import baseline.json

The web viewer shows the span waterfall per run, and lets you select any two runs to see a side-by-side divergence view (/api/diff/{a}/{b} for programmatic access).

Try the demo (offline, no API key)

python examples/research_agent.py   # records two runs with a seeded regression
agentrewind diff <run1> <run2>        # pinpoints the prompt change that caused it

Development

pip install -e '.[dev]'
pytest
ruff check .

Reproducible evaluation

The offline benchmark verifies that replay is byte-for-byte faithful and makes zero provider calls; it also reports local recording and replay throughput:

python benchmarks/replay_benchmark.py --calls 1000

See the evaluation protocol for a repeatable benchmark and CI regression gate workflow, and the architecture overview for implementation details.

Note for macOS: if this repo lives in an iCloud-synced folder (e.g. ~/Documents), create your virtualenv outside it (e.g. ~/.venvs/agentrewind) — the file provider marks files in dot-directories as hidden, and Python ≥3.13 skips hidden .pth files, which breaks editable installs.

MIT licensed.