Skip to main content

llm-replay

Record your LLM calls once. Replay them in tests and CI: deterministic, free, offline. When a prompt changes, see exactly what changed instead of a mystery failure.

pip install pytest-llm-replay

The import and the command stay llm_replay and llm-replay. Only the PyPI name differs, because PyPI treats llm-replay as too close to the existing llmreplay project.

Works with the official OpenAI (and OpenAI-compatible: Groq, Together, vLLM, Ollama), Anthropic and Google Gen AI (Gemini, Vertex AI) Python SDKs, sync and async, streaming included. Nothing to change in your application code.

Why

Refactoring an LLM pipeline raises one question: did the answers change? Live calls are slow, cost money and are not deterministic, so most teams either skip the check or eyeball a few outputs.

llm-replay records each call (request and response) to a small JSONL "tape". Later runs replay from the tape, so:

  • tests and CI run in milliseconds, with no API key and no spend;
  • the same inputs always give the same outputs, so a failing test means your code changed;
  • when a request no longer matches a recording, the failure shows the diff of the request (a reworded prompt, a new temperature, a different tool list);
  • llm-replay diff compares two recordings, e.g. before and after upgrading the model, and lists which answers changed.

The first version of this lived inside a production voice-to-design service, where it proved that a rewrite that cut latency by 31% returned identical results. This is that tool, made general.

Quick start: pytest

import pytest
from myapp import summarise          # uses openai / anthropic / google-genai inside

@pytest.mark.llm_replay
def test_summary_mentions_price():
    assert "price" in summarise("...")
pytest --llm-mode=record      # once, with your API key: writes tests/tapes/<module>__<test>.jsonl
pytest                        # from then on: replays, no key, no network

Commit the tapes. In CI:

- run: pytest                 # default mode is replay
- run: llm-replay check       # fails if any call had no recording, and prints the request diff

Quick start: anywhere

import llm_replay

with llm_replay.session("tapes/eval.jsonl", mode="auto") as s:
    run_my_pipeline()
print(s.report.summary())
# llm-replay: 118 replayed, 2 recorded, 0 missing | 512,330 tokens replayed (~$0.3100 not spent)

Modes

Mode Model called? Tape Use it for
record always rewritten from scratch creating or refreshing a tape
replay never read only; a call with no recording fails tests, CI (default)
auto only for calls not on the tape missing calls are appended growing a test suite
off always untouched turning it off without code changes

Set it with mode=, --llm-mode, the LLM_REPLAY_MODE environment variable, or llm_replay_mode in your pytest ini.

When a recording is missing

ReplayMiss: llm-replay: no recording for openai.chat:gpt-4o-mini (key 817b9ab0ead605d6117c7366).
Closest recorded call is 95% similar; the request changed:
  --- recorded
  +++ this call
  @@ -3,3 +3,3 @@
       {
  -      "content": "Count the words.",
  +      "content": "Count the words. Ignore punctuation.",
         "role": "system"

If the change was intended, re-record (--llm-mode=auto adds only the new calls), review the tape diff, commit.

Command line

llm-replay stats tests/tapes/search.jsonl --prices prices.json   # calls, tokens, what a live run would cost
llm-replay diff old.jsonl new.jsonl --fail-on-change             # answers changed / prompts changed / added / removed
llm-replay check .llm-replay/report.json --max-live 0            # CI gate after a pytest run

prices.json is your own table ({"model": {"input": USD per 1M tokens, "output": ...}}); prices change, so the library ships none. See prices.example.json for the shape.

Your own functions

Anything that isn't one of the supported SDK calls (a raw HTTP call to a model server, an embedding service, a vision endpoint) can be recorded with a decorator:

@llm_replay.replayable("vision.classify")
def classify(image_url: str) -> dict:
    return requests.post(VISION_URL, json={"url": image_url}).json()

Arguments form the key; the return value must be JSON-serialisable (or pass dump= / load=). Outside a session the function runs as usual.

What is recorded, and what isn't

  • Keyed on: model, messages/contents, tools, response format and sampling parameters, after normalisation.
  • Ignored for matching: timeout, user, metadata, extra_headers, request ids and other values that don't change the answer.
  • Never written: API keys, auth headers, HTTP clients. Strings that look like keys (sk-…, sk-ant-…, AIza…, gsk_…, hf_…, bearer tokens) are replaced with [REDACTED]. Tapes are plain text: read them before committing.
  • Repeated identical calls (sampling with temperature > 0) replay in the recorded order.
  • Binary inputs (audio, images) are keyed by their SHA-256 and not stored.

How it compares

HTTP recorders such as vcrpy and pytest-recording replay raw HTTP. They work, but every prompt or parameter change becomes an opaque mismatch, streaming bodies are awkward, and they know nothing about tokens or cost. llm-replay works at the SDK call: it understands requests and responses, tells you what changed, rebuilds real SDK response objects (so type checks and attribute access still work), and reports tokens and cost. If you need to record arbitrary HTTP, use vcrpy; if you test LLM pipelines, this is narrower and more helpful.

Limitations (v0.1)

  • SDK helpers that bypass create are not intercepted yet: OpenAI .parse() / .stream(), Anthropic .stream(). Use create(..., stream=True) or wrap your function with @replayable.
  • A stream is written to the tape when it is fully consumed or closed.
  • Patching is process-wide while a session is open; run sessions one at a time per process (pytest's default).

Development

pip install -e ".[dev]"
pytest
ruff check src tests && ruff format --check src tests
python examples/offline_demo.py      # the whole workflow against a stub model, no key needed

Tests run the real SDK clients against an in-process HTTP stub, so they never touch the network.

License

MIT

Metadata

Release files for pytest-llm-replay 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-llm-replay 0.1.3
File Size Uploaded Provenance Actions
pytest_llm_replay-0.1.3.tar.gz 27.8 kB With provenance Details for pytest_llm_replay-0.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-llm-replay 0.1.3
File Interpreter ABI Platform Actions
pytest_llm_replay-0.1.3-py3-none-any.whl Python 3 none any Details for pytest_llm_replay-0.1.3-py3-none-any.whl

Total release size: 51.4 kB

Release files / pytest_llm_replay-0.1.3.tar.gz

Download URL pytest_llm_replay-0.1.3.tar.gz
Size 27.8 kB
Tags Source
SHA-256 checksum
How to use checksums
cdb1f95ab5bb7d0634ed26161be3948c83e85bdbaff208d646d9416151bbbcb4
BLAKE2b-256 checksum
How to use checksums
eab25e4d7fc4e0db4970688720bd90f04d76dfc1f1843543ac3ae47f5df5a00a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Release files / pytest_llm_replay-0.1.3-py3-none-any.whl

Download URL pytest_llm_replay-0.1.3-py3-none-any.whl
Size 23.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2f3d0450edfb140ab825c2de25edc2d0bc6cbabcd530bf0ce4524895e6bb2a10
BLAKE2b-256 checksum
How to use checksums
376c7323e5c2b73250f2d3a0c47bee5a81ea4100de83f2a2a7311cda382a8b9f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Security for pytest-llm-replay 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pytest_llm_replay-0.1.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.3 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page