AgentRewind

Flight recorder for LLM agents. Trace every LLM and tool call an agent makes, replay a run deterministically (no API calls, no cost, no nondeterminism), and diff two runs to find exactly where their behavior diverged.

"It worked yesterday — why is it different today?" is the defining debugging problem of agent development. Tracing tools show you what happened; AgentRewind also lets you re-execute what happened and compare runs structurally.

Install

pip install llm-run-recorder              # core (stdlib-only)
pip install 'llm-run-recorder[server]'    # + web trace viewer
pip install 'llm-run-recorder[postgres]'  # + PostgreSQL storage backend (psycopg 3)

Trace

import agentrewind as al

@al.traced(kind="tool")
def search(query: str) -> str:
    ...

with al.trace("research-agent"):
    plan = call_llm(...)          # record with al.record_llm_call(...) or a Recorder
    evidence = search(plan.query)

Traces (span tree, inputs/outputs, latency, token counts, errors) are stored in a local SQLite db at ~/.agentrewind/traces.db — no account, no server required.

PostgreSQL backend (optional)

SQLite keeps traces on one machine. PostgreSQL gives a team or a fleet one shared trace store: agents, eval workers and CI jobs on different machines all record into the same database. Any teammate can then list, show, diff or open the viewer on any run, wherever it was recorded, and query every run with ordinary SQL (for example, error rates or token usage by model across all of last week's runs). Choose it for sharing and querying, not speed. On one machine the SQLite default wrote faster in our benchmark.

pip install 'llm-run-recorder[postgres]'
export AGENTREWIND_DB_URL=postgresql://user:password@host:5432/dbname

Everything else stays the same: recording, replay, diff, the CLI and the web viewer. The schema is created on first use. You can also pass the URL explicitly with agentrewind --db postgresql://… list, al.configure(db_path="postgresql://…") or agentrewind.open_store("postgresql://…"). To use a schema other than public, append ?options=-csearch_path%3Dmyschema. Both backends store the same schema and return the same results. See docs/ARCHITECTURE.md for the concurrency model and docs/EVALUATION.md for the concurrent-writer benchmark.

Upgrading from 0.2.x: an existing traces.db is migrated in place the first time 0.3 opens it. The migration is one-way, so 0.2.x cannot write to the file afterwards.

Sensitive-data controls

For production traffic, opt into recursive persistence-time redaction. It removes common credential fields (api_key, authorization, password, secret, and token) and common token formats from trace metadata, span payloads, and the local replay cache:

import agentrewind as al

al.configure(redaction=al.RedactionPolicy())

Redaction intentionally also applies to cached responses, so use it where protecting local data matters more than replaying the original secret verbatim.

Replay

Wrap your provider call in a Recorder. Responses are cached by a canonical request fingerprint; in replay mode the whole agent run re-executes deterministically and offline:

from agentrewind.replay import Recorder

llm = Recorder(call_openai, mode="record")   # live run, responses cached
llm = Recorder(call_openai, mode="replay")   # deterministic re-run, zero API calls
llm = Recorder(call_openai, mode="auto")     # replay on hit, record on miss

Or instrument an SDK client in place — existing code keeps calling it exactly as before:

import agentrewind as al

client = al.instrument(OpenAI(), mode="auto")       # or Anthropic()
client.chat.completions.create(model="gpt-4o", messages=[...])  # traced + replayable

(Prefer explicit wrappers? agentrewind.providers.OpenAIChat / AnthropicMessages do the same without monkey-patching.)

Streaming is captured too: Recorder.call_stream / acall_stream pass chunks through live while recording them, and replay re-streams the identical chunks offline.

Requests containing volatile fields (timestamps, request ids) can be normalized before fingerprinting with Recorder(..., canonicalize=strip_volatile) so they still replay.

Async agents are supported throughout: @al.traced works on async def functions (context propagates through awaits and asyncio.gather), and Recorder.acall is the awaitable variant for async provider clients.

Diff

$ agentrewind diff 3f2a91 8c17d0
2 divergence(s); first divergence is where the runs split:

1. [input] /llm:mock-4o — inputs differ
     left : {"messages":[{"content":"decide next step",...
     right: {"messages":[{"content":"decide the next step",...
2. [output] /tool:search — outputs differ

The diff walks both span trees in parallel and reports structure, input, and output divergences in execution order — entry #1 is where the runs first split.

CLI & viewer

agentrewind list                 # recent runs
agentrewind show <trace-id>      # span tree with latencies
agentrewind diff <run1> <run2>   # structural diff (exit code 2 if runs differ)
agentrewind serve                # web viewer at http://127.0.0.1:4317
agentrewind export <trace-id> -o baseline.json  # portable artifact for a bug report or CI fixture
agentrewind import baseline.json

The web viewer shows the span waterfall per run, and lets you select any two runs to see a side-by-side divergence view (/api/diff/{a}/{b} for programmatic access).

Try the demo (offline, no API key)

python examples/research_agent.py   # records two runs with a seeded regression
agentrewind diff <run1> <run2>        # pinpoints the prompt change that caused it

Development

pip install -e '.[dev]'
pytest                # SQLite half of the suite; PostgreSQL tests skip without a database
ruff check .

Every test is parametrized over both storage backends. To run the full suite locally, start the bundled PostgreSQL first:

docker compose up -d
export AGENTREWIND_DB_URL=postgresql://agentrewind:agentrewind@localhost:55432/agentrewind
pytest --cov-fail-under=97

296 tests (148 per backend), 99.57% branch coverage with PostgreSQL available. CI runs the full suite against a PostgreSQL 17 service container on Python 3.10, 3.12 and 3.13, and enforces the 97% coverage floor. The floor is CI-only because a local run without PostgreSQL leaves the backend's module uncovered.

The suites worth knowing about:

• test_diff_alignment.py pins the greedy span-alignment walk in diff.py. When two runs take different shapes, that walk decides whether an extra step is reported against the left or the right run, which is what makes the first divergence point at the real cause. Covers insertions, deletions, positional swaps, trailing spans and same-name-different-kind.
• test_replay.py and test_streaming.py check that a replayed run is byte-for-byte identical to the recording and makes no provider calls.
• test_concurrency.py and test_async.py cover concurrent recording into one store.
• test_redaction.py covers credential scrubbing before anything reaches the store.
• test_storage_backends.py is the backend contract: identical ordering, lookup and round-trip behaviour on both backends; migrating a database written by the published 0.2.2 release (tests/fixtures/traces_v0_2_2.db); and separate writer processes saving distinct traces and contending for the same one.

Dependency updates come through Renovate (renovate.json); GitHub Actions bumps are grouped and automerged once CI is green, majors always get a human review.

Reproducible evaluation

The offline benchmark verifies that replay is byte-for-byte faithful and makes zero provider calls; it also reports local recording and replay throughput:

python benchmarks/replay_benchmark.py --calls 1000

A second benchmark measures 1, 4, 8, 16 and 32 concurrent writer processes on each storage backend (traces/sec and p95 write latency):

python benchmarks/storage_benchmark.py --output results.json

See the evaluation protocol for a repeatable benchmark and CI regression gate workflow, and the architecture overview for implementation details.

Note for macOS: if this repo lives in an iCloud-synced folder (e.g. ~/Documents), create your virtualenv outside it (e.g. ~/.venvs/agentrewind) — the file provider marks files in dot-directories as hidden, and Python ≥3.13 skips hidden .pth files, which breaks editable installs.

MIT licensed.