llmvcr 0.1.1

Record and replay LLM API responses for fast, free, deterministic tests

llmvcr

Record and replay LLM API responses for fast, free, deterministic tests.

---

The Problem

Testing code that calls an LLM API is painful:

Problem	Impact
🐢 Slow	Each test waits 2–10 seconds for a real API response
💸 Expensive	Every test run costs real money
🎲 Flaky	Same prompt, different response — random test failures
📡 Needs internet	Tests break offline or when the API is down
🔒 Secrets in CI	You must inject API keys into every environment

llmvcr solves all five by recording real LLM responses once and replaying them instantly — for free, offline, with 100% identical results every time.

---

How It Works

Just like VCR.py records and replays HTTP requests, llmvcr records and replays LLM API responses. But unlike VCR.py, it works at the SDK layer — not the raw HTTP layer — so it handles streaming, never touches your API key, and produces clean human-readable YAML cassettes.

First run   →  hits real API  →  saves response to cassette YAML
Every run after  →  reads from cassette  →  instant, free, offline

---

Supported Providers

Provider	SDK	Models	Install
OpenAI	openai	GPT-4, GPT-4o, GPT-3.5	pip install llmvcr[openai]
Anthropic	anthropic	Claude 3.5, Claude 3	pip install llmvcr[anthropic]
Google Gemini	google-genai	Gemini 2.0, 1.5	pip install llmvcr[gemini]
Llama (local)	ollama	Llama 3.1, 3.2, Mistral, Qwen	pip install llmvcr[ollama]
Llama (cloud)	groq	Llama 3.3 70B, 3.1 8B	pip install llmvcr[groq]

---

Installation

# Core + one provider
pip install llmvcr[openai]
pip install llmvcr[anthropic]
pip install llmvcr[gemini]
pip install llmvcr[ollama]
pip install llmvcr[groq]

# Everything at once
pip install llmvcr[all]

Ollama note: Also requires the Ollama app running locally. Download from ollama.com, then ollama pull llama3.1.

---

Quick Start

1. The @cassette decorator (recommended for tests)

Add one decorator to your test. First run records, every run after replays.

import llmvcr
import openai

@llmvcr.cassette("cassettes/summarize.yaml")
def test_summarize():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Summarize AI in one sentence."}]
    )
    assert "intelligence" in response.choices[0].message.content

# First run:  hits OpenAI API → saves to cassettes/summarize.yaml
# Every run:  reads from cassette → instant, free, offline

2. Context managers (for scripts and notebooks)

import llmvcr
import openai

# Record
with llmvcr.record("cassettes/demo.yaml"):
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hello!"}]
    )

# Replay (never hits the API)
with llmvcr.playback("cassettes/demo.yaml"):
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    print(response.choices[0].message.content)  # instant

3. Explicit modes

# auto (default) — record if cassette missing, replay if it exists
@llmvcr.cassette("cassettes/test.yaml")

# record — always re-record, overwrite cassette
@llmvcr.cassette("cassettes/test.yaml", mode="record")

# playback — always replay, raise error if cassette missing
@llmvcr.cassette("cassettes/test.yaml", mode="playback")

---

Provider Examples

OpenAI

import llmvcr
import openai

@llmvcr.cassette("cassettes/openai_test.yaml", provider="openai")
def test_openai():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "What is Python?"}]
    )
    assert "programming" in response.choices[0].message.content

Anthropic

import llmvcr
import anthropic

@llmvcr.cassette("cassettes/claude_test.yaml", provider="anthropic")
def test_anthropic():
    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        messages=[{"role": "user", "content": "What is Python?"}]
    )
    assert "programming" in response.content[0].text

Google Gemini

import llmvcr
from google import genai
from google.genai import types

@llmvcr.cassette("cassettes/gemini_test.yaml", provider="gemini")
def test_gemini():
    client = genai.Client()
    response = client.models.generate_content(
        model="gemini-2.0-flash",
        contents="What is Python?",
        config=types.GenerateContentConfig(
            system_instruction="You are a concise assistant."
        )
    )
    assert "programming" in response.text

Llama via Ollama (local)

import llmvcr
import ollama

@llmvcr.cassette("cassettes/llama_local_test.yaml", provider="ollama")
def test_ollama():
    response = ollama.chat(
        model="llama3.1",
        messages=[{"role": "user", "content": "What is Python?"}]
    )
    assert "programming" in response.message.content
    # also works: response["message"]["content"]

Llama via Groq (cloud)

import llmvcr
from groq import Groq

@llmvcr.cassette("cassettes/llama_groq_test.yaml", provider="groq")
def test_groq():
    client = Groq()
    response = client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": "What is Python?"}]
    )
    assert "programming" in response.choices[0].message.content

---

What a Cassette Looks Like

After recording, a clean human-readable YAML file is saved — no binary blobs, no API keys, safe to commit to git:

llmvcr_version: '0.1'
provider: openai
recorded_at: '2025-03-03T10:24:00Z'
interactions:
  - request:
      model: gpt-4
      messages:
        - role: user
          content: Summarize AI in one sentence.
    response:
      id: chatcmpl-abc123
      choices:
        - message:
            role: assistant
            content: Artificial intelligence is the simulation of human-like reasoning
              and decision-making by computer systems.
      usage:
        prompt_tokens: 14
        completion_tokens: 22
        total_tokens: 36

You can review it, edit it, and diff it in pull requests just like any other file.

---

CLI

# Inspect a cassette — see stored interactions, token counts, models
llmvcr info cassettes/demo.yaml

# Record a script to a cassette
llmvcr record --output cassettes/demo.yaml python my_script.py

# Replay a script using a cassette (no API call made)
llmvcr playback --cassette cassettes/demo.yaml python my_script.py

Example llmvcr info output:

Cassette:      cassettes/demo.yaml
Provider:      openai
Recorded at:   2025-03-03T10:24:00Z
Version:       0.1
Interactions:  2

  [1] model=gpt-4
      prompt:   'Summarize AI in one sentence.'
      response: 'Artificial intelligence is the simulation of...'
      tokens:   36

  [2] model=gpt-4
      prompt:   'What is Python?'
      response: 'Python is a high-level programming language...'
      tokens:   38

---

Why Not VCR.py or pytest-recording?

Those tools were built for generic HTTP APIs. LLMs break them in several ways:

Issue	VCR.py / pytest-recording	llmvcr
Streaming responses	❌ Crashes with chunked read errors	✅ Records and replays natively
API key in cassette	❌ Saved in plaintext by default	✅ Never sees credentials
Adding temperature=0.7	❌ Breaks cassette match	✅ Ignored, match still works
Gzip-encoded responses	❌ Saves as unreadable binary	✅ Always clean YAML
Zero-config setup	❌ Requires conftest.py boilerplate	✅ Works out of the box

The root cause: VCR.py intercepts raw HTTP bytes. llmvcr intercepts at the SDK layer, where responses are already decoded Python objects — no gzip, no auth headers, no socket state.

---

Limitations

• Changing prompts, message structure, provider, or model may require re-recording a cassette.
• Request matching is based on the serialized request content used by the provider integration.
• Provider SDK changes may require updates in future llmvcr releases.
• Local-provider workflows like Ollama require the model/runtime to be available when recording.

---

Running the Tests

Install dev dependencies

# Clone the repo
git clone https://github.com/shahid-43/llmvcr
cd llmvcr

# Install in editable mode with dev dependencies
pip install -e ".[all]"
pip install pytest

Run all tests

pytest

Run with verbose output (see each test name)

pytest -v

Run a specific test file

pytest tests/test_matching.py
pytest tests/test_storage.py
pytest tests/test_recorder.py
pytest tests/test_cassettes.py
pytest tests/test_new_providers.py

Run a specific test by name

pytest -v -k "test_playback_ignores_temperature"

Run with coverage report

pip install pytest-cov
pytest --cov=llmvcr --cov-report=term-missing

Expected output

============================= test session starts ==============================
collected 81 items

tests/test_cassettes.py ....................                             [ 25%]
tests/test_matching.py .........                                        [ 36%]
tests/test_new_providers.py ...................................           [ 79%]
tests/test_recorder.py .........                                        [ 90%]
tests/test_storage.py .......                                           [100%]

============================== 81 passed in 0.70s ==============================

All 81 tests run without any API keys — they use pre-recorded cassettes in tests/cassettes/.

---

Project Structure

llmvcr/
├── pyproject.toml                  # Package metadata and build config
├── requirements.txt                # All dependencies listed
├── README.md
├── LICENSE
│
├── src/
│   └── llmvcr/
│       ├── __init__.py             # Public API: cassette, record, playback
│       ├── cassette.py             # @llmvcr.cassette() decorator
│       ├── context.py              # with llmvcr.record() / playback()
│       ├── recorder.py             # Core engine: record/playback state
│       ├── matching.py             # Semantic request matching
│       ├── storage.py              # Read/write cassette YAML files
│       ├── errors.py               # Custom exceptions
│       ├── _patch.py               # Provider routing
│       ├── cli.py                  # CLI commands
│       │
│       └── providers/
│           ├── openai_provider.py
│           ├── anthropic_provider.py
│           ├── gemini_provider.py
│           ├── ollama_provider.py
│           └── groq_provider.py
│
└── tests/
    ├── cassettes/
    │   ├── sample_openai.yaml
    │   ├── sample_anthropic.yaml
    │   ├── sample_gemini.yaml
    │   ├── sample_ollama.yaml
    │   └── sample_groq.yaml
    ├── test_matching.py
    ├── test_storage.py
    ├── test_recorder.py
    ├── test_cassettes.py
    └── test_new_providers.py

---

Error Reference

Error	Cause	Fix
CassetteNotFoundError	Used mode="playback" but no cassette file exists	Run once with mode="record" first
NoMatchFoundError	Request doesn't match any interaction in the cassette	Re-record the cassette or add the interaction
ProviderNotSupportedError	Unknown provider name passed	Use one of: openai, anthropic, gemini, ollama, groq

---

Contributing

1. Fork the repo
2. Create a branch: git checkout -b my-feature
3. Make your changes and add tests
4. Run the test suite: pytest
5. Open a pull request

When adding a new provider, follow the pattern in src/llmvcr/providers/openai_provider.py — implement serialize_request, serialize_response, deserialize_response, and make_patch, then register the provider in _patch.py.

---

License

This project uses the MIT LICENSE.

That means other developers can use, modify, publish, and even use your package commercially, as long as they keep the license notice and copyright text.

Contributing

1. Fork the repo
2. Create a branch
3. Add tests for changes
4. Run pytest -v
5. Open a pull request