replayai-sdk 0.5.4

ReplayAI Python SDK — record, replay, and debug AI agent executions. Instrument agents with a single decorator, scrub timelines, diff runs, export regression tests.

replayai — Python SDK

Instrument Python agents, record every step (LLM calls, tool calls, retrievals, errors), and POST sessions to the ReplayAI dashboard where you can replay them, diff runs, and export tests.

• Stdlib only — pip install replayai-sdk brings no dependencies.
• Decorator + context manager — @trace(...) or with trace(...) as ctx:.
• Framework extras — pip install "replayai-sdk[langchain]" for auto-instrumentation.

Install

pip install replayai-sdk
# or, with the LangChain integration:
pip install "replayai-sdk[langchain]"

Windows: CLI not found after install?

On Windows, pip installs the replayai CLI to %APPDATA%\Python\Python3XX\Scripts\ which may not be on your PATH. Fix it by adding that directory to your PATH:

# Add to PATH for the current session:
$env:PATH += ";$env:APPDATA\Python\Scripts"

# Or add permanently:
[Environment]::SetEnvironmentVariable("PATH", $env:PATH + ";$env:APPDATA\Python\Scripts", "User")

# Then verify:
replayai --help

Alternatively, invoke the CLI via python -m:

python -m replayai.cli --help
python -m replayai.cli record agent.py --project my-project
python -m replayai.cli test tests/replay/
python -m replayai.cli ui --port 3000

30-second usage

from replayai import trace, record_step

@trace("support-agent-v3", project="support-agent", tags=["production"])
def handle_support_ticket(message: str) -> str:
    record_step(
        type="llm_call", name="classify_intent",
        model="gpt-4o-mini", tokens_in=312, tokens_out=24,
        input=f"User: {message}", output="intent: billing_dispute",
        status="success",
    )
    record_step(
        type="tool_call", name="issue_refund",
        input='{"charge_id":"ch_002"}',
        output='{"refund_id":"ref_3391"}',
        status="success",
    )
    return "Refund issued (ref_3391)."

handle_support_ticket("I was charged twice, refund me.")

Open the dashboard — your run is there with a full timeline.

Configuration

Environment variables (all optional):

Variable	Default	Description
REPLAYAI_PROJECT	—	Default project slug/id
REPLAYAI_TOKEN	—	Cloud API token
REPLAYAI_STORAGE	cloud	cloud, local, or both
REPLAYAI_API_URL	http://localhost:3000	Cloud API base URL
REPLAYAI_SAMPLE_RATE	1.0	Fraction of sessions to record
REPLAYAI_STRICT	false	Raise on recording failures
REPLAYAI_REDACT_PATTERNS	built-in set	Comma-separated regexes

Programmatic override:

import replayai
replayai.configure(project="support-agent", api_url="http://localhost:3000")
replayai.strict_mode = True  # opt into hard failures

Async

import asyncio
from replayai import atrace, arecord_step

@atrace("async-agent")
async def handle(message: str) -> str:
    await arecord_step(type="llm_call", name="classify", status="success")
    return "ok"

asyncio.run(handle("hello"))

Replay & export

from replayai import ReplaySession

replay = ReplaySession("ses_8fa1")

# Register mocks with flexible matching:
replay.mock("issue_refund", '{"refund_id":"ref_3391"}')
replay.mock("search_web", response, is_prefix=True)  # prefix match
replay.mock(r"search_web\(.*\)", response, is_regex=True)  # regex match
replay.mock("tool", response, input_contains="NYC")  # input-contains match

# load() fetches the recorded session (does NOT re-execute agent code)
trace_obj = replay.load()
print(trace_obj.step_count, trace_obj.status)

# compare() runs your agent function with mocks applied, then diffs vs loaded session
result = replay.compare(my_agent_fn, inputs="user message")
print(result.matches, result.divergences)

# run() is deprecated — use load() instead
# trace_obj = replay.run(agent="...", framework="...")  # ← deprecated

# Export as pytest or jest test
print(replay.export(lang="pytest"))

CLI

# Record a script
replayai record agent.py --project my-project --tags prod

# Run replay regression tests
replayai test tests/replay/ --tb=short

# Start the dashboard (no token required for local dev)
replayai ui --port 3000

Windows: CLI not on PATH?

# Use python -m as a fallback (no PATH needed)
python -m replayai ui --port 3000
python -m replayai record agent.py --project my-project

LangChain integration

from replayai.integrations.langchain import trace_agent, ReplayCallbackHandler

@trace_agent("support-agent-v3", project="support-agent", tags=["production"])
def handle(message: str) -> str:
    return executor.invoke({"input": message})["output"]

See examples/quickstart.py and examples/langchain_demo.py for runnable demos.

License

MIT.