llm-vcr

Record and replay LLM HTTP traffic for deterministic, key-free pytest runs.

Status: v0.6 — SSE streaming replay, sequential tool-call cassettes, OpenAI + Anthropic semantic matching, model-date normalize, and llm-vcr diff.

60-second try

pip install pytest-llm-vcr
llm-vcr health
docker compose run --rm test    # cassette replay, no API key

Why this vs alternatives

Problem

Testing code that calls LLMs is slow, flaky, and expensive. Hand-written mocks drift from reality. Generic HTTP cassettes (VCR.py) don't understand LLM request shapes or redact API keys well.

Key features (v0.6)

• pytest plugin — @llm_vcr decorator, llm_vcr_client fixture, --llm-vcr-record
• httpx transport — sync + async, including client.stream(...)
• YAML cassettes — streaming: true + chunks for SSE
• Matching — exact (default) or matcher="semantic" (volatile keys, model aliases, messages by role+content, tools by name); Anthropic Messages API via api.anthropic.com URL detection
• llm-vcr diff — show normalized differences between JSON bodies or cassette interactions
• Automatic redaction — strips api_key, x-api-key, token, authorization fields

Architecture

@pytest test
    └── @llm_vcr decorator
            └── VCRTransport (httpx)
                    ├── replay mode → read cassette YAML
                    └── record mode → live HTTP + save cassette

Installation

pip install pytest-llm-vcr
pip install -e ".[dev]"  # from source

Local development

pip install -e ".[dev]"
pytest tests/ -v
llm-vcr health
llm-vcr diff left.json right.json

Docker

docker compose run --rm test
docker compose run --rm health

Configuration

Usage

Replay (CI — no API key needed)

import httpx
from llm_vcr.plugin import llm_vcr

@llm_vcr("my_test")
def test_chat(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.openai.com/v1/chat/completions",
        json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}]},
    )
    assert resp.json()["choices"][0]["message"]["content"]

Record new cassettes

LLM_VCR_RECORD=true pytest tests/ --llm-vcr-record

Diff normalized request bodies

llm-vcr diff left.json right.json
llm-vcr diff cassette_a.yaml cassette_b.yaml --index 0

Model strings with a trailing -YYYY-MM-DD normalize equal so dated aliases do not show as diffs.

Anthropic Messages API

Point httpx at https://api.anthropic.com/v1/messages the same way as OpenAI — record once, replay in CI without a key. Query strings (e.g. beta flags) are stripped for matching. Use matcher="semantic" to ignore metadata and tool_use / tool_result ids:

@llm_vcr("anthropic_chat", matcher="semantic")
def test_anthropic(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.anthropic.com/v1/messages",
        json={
            "model": "claude-3-5-sonnet",
            "max_tokens": 64,
            "messages": [{"role": "user", "content": "Say hello"}],
        },
        headers={"x-api-key": "sk-ant-...", "anthropic-version": "2023-06-01"},
    )
    assert resp.json()["content"][0]["text"]

x-api-key is redacted in cassette bodies when present.

Running tests

pytest tests/ -v

Roadmap

• SSE streaming chunk replay
• Tool-call multi-step loops (@llm_vcr(..., sequential=True))
• Async httpx transport (fixture + replay)
• Model-date normalize + llm-vcr diff
• OpenAI + Anthropic semantic request matching (matcher="semantic")

Known limitations (v0.6)

• Sequential matching is opt-in (sequential=True); default matching is still hash-based
• Matching: exact hash (default) or semantic (volatile keys, model aliases, message/tool shape; Anthropic when host is api.anthropic.com)
• Record mode for streaming stores chunks, not per-event timestamps

License

MIT