llm-vcr

Record and replay LLM HTTP traffic for deterministic, key-free pytest runs.

Status: v0.6 — SSE streaming replay, sequential tool-call cassettes, OpenAI + Anthropic semantic matching, model-date normalize, and llm-vcr diff.

60-second try

pip install pytest-llm-vcr
llm-vcr health
docker compose run --rm test    # cassette replay, no API key

Why this vs alternatives

Approach	Strength	Gap
llm-vcr	LLM-aware matching, redaction, SSE + sequential cassettes	httpx-focused today
VCR.py / pytest-recording	Mature generic HTTP cassettes	No model-date normalize or LLM-shaped diffs
Hand-written mocks	Fast, no network	Drift from live provider payloads
Live API in CI	Highest fidelity	Flaky, keyed, and expensive

Problem

Testing code that calls LLMs is slow, flaky, and expensive. Hand-written mocks drift from reality. Generic HTTP cassettes (VCR.py) don't understand LLM request shapes or redact API keys well.

Key features (v0.6)

• pytest plugin — @llm_vcr decorator, llm_vcr_client fixture, --llm-vcr-record
• httpx transport — sync + async, including client.stream(...)
• YAML cassettes — streaming: true + chunks for SSE
• Matching — exact (default) or matcher="semantic" (volatile keys, model aliases, messages by role+content, tools by name); Anthropic Messages API via api.anthropic.com URL detection
• llm-vcr diff — show normalized differences between JSON bodies or cassette interactions
• Automatic redaction — strips api_key, x-api-key, token, authorization fields

Architecture

@pytest test
    └── @llm_vcr decorator
            └── VCRTransport (httpx)
                    ├── replay mode → read cassette YAML
                    └── record mode → live HTTP + save cassette

Component	Technology	Why
HTTP	httpx	Modern, sync+async, transport hooks
Cassettes	YAML	Readable diffs in PRs
Tests	pytest entry point	Zero-config discovery

Installation

pip install pytest-llm-vcr
pip install -e ".[dev]"  # from source

Local development

pip install -e ".[dev]"
pytest tests/ -v
llm-vcr health
llm-vcr diff left.json right.json

Docker

docker compose run --rm test
docker compose run --rm health

Configuration

Variable	Default	Description
LLM_VCR_CASSETTE_DIR	tests/cassettes	Cassette storage directory
LLM_VCR_RECORD	false	Force record mode

Usage

Replay (CI — no API key needed)

import httpx
from llm_vcr.plugin import llm_vcr

@llm_vcr("my_test")
def test_chat(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.openai.com/v1/chat/completions",
        json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}]},
    )
    assert resp.json()["choices"][0]["message"]["content"]

Record new cassettes

LLM_VCR_RECORD=true pytest tests/ --llm-vcr-record

Diff normalized request bodies

llm-vcr diff left.json right.json
llm-vcr diff cassette_a.yaml cassette_b.yaml --index 0

Model strings with a trailing -YYYY-MM-DD normalize equal so dated aliases do not show as diffs.

Anthropic Messages API

Point httpx at https://api.anthropic.com/v1/messages the same way as OpenAI — record once, replay in CI without a key. Query strings (e.g. beta flags) are stripped for matching. Use matcher="semantic" to ignore metadata and tool_use / tool_result ids:

@llm_vcr("anthropic_chat", matcher="semantic")
def test_anthropic(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.anthropic.com/v1/messages",
        json={
            "model": "claude-3-5-sonnet",
            "max_tokens": 64,
            "messages": [{"role": "user", "content": "Say hello"}],
        },
        headers={"x-api-key": "sk-ant-...", "anthropic-version": "2023-06-01"},
    )
    assert resp.json()["content"][0]["text"]

x-api-key is redacted in cassette bodies when present.

Running tests

pytest tests/ -v

Roadmap

• SSE streaming chunk replay
• Tool-call multi-step loops (@llm_vcr(..., sequential=True))
• Async httpx transport (fixture + replay)
• Model-date normalize + llm-vcr diff
• OpenAI + Anthropic semantic request matching (matcher="semantic")

Known limitations (v0.6)

• Sequential matching is opt-in (sequential=True); default matching is still hash-based
• Matching: exact hash (default) or semantic (volatile keys, model aliases, message/tool shape; Anthropic when host is api.anthropic.com)
• Record mode for streaming stores chunks, not per-event timestamps

License

MIT