llmreplay 0.1.1

Deterministic replay debugger for LLM agents

llmreplay

Deterministic replay debugger for LLM agents. Records LLM calls and tool executions to SQLite, replays from the log with no network calls.

from llmreplay import record, replay

# Record
with record("my_run", seed=42):
    response = client.chat.completions.create(...)

# Replay — zero network calls
session = replay("my_run")
for event in session.events():
    print(event.step, event.kind, event.payload)

Install

pip install llmreplay

Requirements: Python >= 3.10

What gets recorded

• LLM requests/responses (OpenAI, Anthropic, Grok/xAI, Gemini)
• Tool calls/results (via @record_tool decorator)
• Random seeds (Python random, numpy)
• Exceptions

Events are stored in ~/.llmreplay/<run_id>.db.

CLI

llmreplay list                    # List recorded runs
llmreplay view my_run             # Show all events
llmreplay view my_run --step 42   # Jump to step
llmreplay cost my_run             # Cost breakdown
llmreplay export my_run --json    # Export bug report
llmreplay web my_run              # Launch timeline UI

Features

Auto-instrumentation — OpenAI, Anthropic, Grok, Gemini, LangChain hooks install automatically within record() context.

Tool mocking — Record tool I/O with @record_tool, replay with ToolMocker:

from llmreplay import ToolMocker, EventStore

mocker = ToolMocker()
mocker.load(EventStore("my_run"))

@mocker.mock(name="fetch_price")
def fetch_price(ticker: str) -> dict: ...  # returns recorded result

Regression testing — Run recorded traces against updated code:

from llmreplay import RegressionSuite

suite = RegressionSuite()

@suite.case("run_001")
def check(original, session):
    return session.total_cost() <= original["total_cost_usd"] * 1.1

suite.run()

Fork/branch — Copy a trace up to a step for counterfactual debugging:

from llmreplay import fork
new_store = fork("broken_run", "fixed_run", at_step=50)

Fine-tuning export — Export prompt/response pairs:

from llmreplay import export_finetune_dataset
export_finetune_dataset(["run_001", "run_002"], "data.jsonl")

License

MIT