stubllm 0.1.0

Deterministic mock server for LLM APIs. Test your AI code without spending tokens.

stubllm

Deterministic mock server for LLM APIs. Test your AI code without spending tokens.

Works with: OpenAI · Anthropic · Google Gemini

---

30-second quickstart

# 1. Install
pip install stubllm

# 2. Create a fixture
mkdir fixtures
cat > fixtures/chat.yaml << 'EOF'
fixtures:
  - name: "greeting"
    match:
      provider: openai
      messages:
        - role: user
          content:
            contains: "hello"
    response:
      content: "Hello! How can I help you today?"
EOF

# 3. Start the server
stubllm serve --port 8765

# 4. Point your code at it
export OPENAI_BASE_URL=http://localhost:8765/v1/
python your_app.py  # no real API calls, no tokens spent

---

Why stubllm?

	stubllm	Real API	Ollama
Cost	Free	Paid	Free
Speed	<1ms	1-30s	5-30s
Deterministic	✅	❌	❌
Works offline	✅	❌	✅
No GPU needed	✅	✅	❌
Pytest integration	✅	❌	❌
Fixtures / record-replay	✅	❌	❌
CI-friendly	✅	Slow/expensive	Heavy

---

Installation

pip install stubllm

# With pytest support
pip install "stubllm[pytest]"

---

Fixture format

Fixtures are YAML (or JSON) files that map request patterns to responses.

Basic text response

fixtures:
  - name: "greeting"
    match:
      provider: openai          # openai | anthropic | gemini | any
      endpoint: /v1/chat/completions   # optional
      model: "gpt-4o"           # optional
      messages:
        - role: user
          content:
            contains: "hello"   # exact | contains | regex
    response:
      content: "Hello! How can I help you today?"
      usage:
        prompt_tokens: 10
        completion_tokens: 12
        total_tokens: 22

Tool call response

fixtures:
  - name: "weather_tool"
    match:
      provider: openai
      messages:
        - role: user
          content:
            contains: "weather"
      tools_present: true        # only match when tools are provided
    response:
      tool_calls:
        - id: "call_abc123"
          type: function
          function:
            name: "get_weather"
            arguments: '{"location": "Amsterdam"}'

Streaming with delay

fixtures:
  - name: "slow_story"
    match:
      messages:
        - role: user
          content:
            contains: "story"
    response:
      content: "Once upon a time..."
      stream_chunk_delay_ms: 50   # simulate realistic streaming speed

Error response

fixtures:
  - name: "rate_limit"
    match:
      messages:
        - role: user
          content:
            contains: "trigger_error"
    response:
      content: '{"error": {"message": "Rate limit exceeded", "type": "rate_limit_error"}}'
      http_status: 429

Content match strategies

# Exact match (highest priority)
content:
  exact: "Hello, world!"

# Substring match (case-insensitive)
content:
  contains: "weather"

# Regular expression
content:
  regex: "tell me.*joke"

Matching priority

Higher specificity = higher priority. When multiple fixtures match, the most specific wins:

1. exact message content (score: +10)
2. contains message content (score: +5)
3. regex message content (score: +4)
4. model specified (score: +2)
5. tools_present specified (score: +2)
6. provider specified (score: +1)
7. Fallback (no match criteria)

---

Pytest plugin

Basic setup

# conftest.py — nothing needed, stubllm auto-registers as a pytest plugin
# The `stubllm_server` fixture is available automatically after installing stubllm

# test_my_app.py
import openai
from stubllm.pytest_plugin import use_fixtures

@use_fixtures("fixtures/chat.yaml")
def test_greeting(stubllm_server):
    client = openai.OpenAI(
        base_url=stubllm_server.openai_url,  # includes /v1/ — openai SDK needs this
        api_key="test-key"
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "hello world"}]
    )
    assert "Hello" in response.choices[0].message.content
    assert stubllm_server.call_count == 1

Assertion helpers

def test_with_assertions(stubllm_server):
    # ... make some calls ...

    # Assert specific prompt was sent
    stubllm_server.assert_called_with_prompt("hello")

    # Assert number of calls
    assert stubllm_server.call_count == 2

    # Inspect all calls
    for call in stubllm_server.calls:
        print(call["path"], call["body"])

    # Assert last call path
    stubllm_server.assert_last_call_path("/v1/chat/completions")

Multiple fixture files

@use_fixtures("fixtures/chat.yaml", "fixtures/tools.yaml")
def test_combined(stubllm_server):
    ...  # both fixture files are active for this test

---

Multi-provider support

OpenAI

import openai

client = openai.OpenAI(
    base_url="http://localhost:8765/v1/",  # note: /v1/ required — the OpenAI SDK does not add it
    api_key="test-key"
)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "hello"}]
)

Anthropic

import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:8765",  # Anthropic SDK adds /v1/ itself
    api_key="test-key"
)
message = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "hello"}]
)

Google Gemini

import google.generativeai as genai

genai.configure(
    api_key="test-key",
    client_options={"api_endpoint": "localhost:8765"}
)
model = genai.GenerativeModel("gemini-pro")
response = model.generate_content("hello")

---

Streaming

All providers support streaming. Fixtures work identically — streaming is controlled by the stream: true parameter in the request, not the fixture.

# OpenAI streaming
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content, end="", flush=True)

Control streaming speed in fixtures:

response:
  content: "A long streaming response..."
  stream_chunk_delay_ms: 20   # default: 20ms between chunks

---

Record and replay

Record real API interactions for later replay:

# Start in record mode (proxies to real OpenAI, saves fixtures)
stubllm record \
  --target https://api.openai.com \
  --fixture-dir ./recorded_fixtures

# Run your app against the recording proxy
OPENAI_BASE_URL=http://localhost:8765/v1/ python your_app.py

# Fixtures are saved to ./recorded_fixtures/
ls recorded_fixtures/
# recorded_hello_world_1706000000.yaml
# recorded_weather_query_1706000001.yaml

Recorded fixtures are sanitized (API keys removed) and can be committed to your repo.

---

CLI reference

# Start server (auto-loads ./fixtures/ if it exists)
stubllm serve

# Custom port and fixture directory
stubllm serve --port 9000 --fixture-dir ./my-fixtures

# Multiple fixture directories
stubllm serve --fixture-dir ./fixtures/openai --fixture-dir ./fixtures/anthropic

# Individual fixture files
stubllm serve --fixture-file chat.yaml --fixture-file tools.yaml

# Record mode
stubllm record --target https://api.openai.com --fixture-dir ./recorded

# Version
stubllm --version

---

Structured output (JSON schema)

When response_format: { type: "json_schema" } is set, stubllm validates that the fixture response is valid JSON. If it's not, it wraps the content automatically.

fixtures:
  - name: "structured"
    match:
      provider: openai
    response:
      content: '{"name": "Alice", "age": 30}'  # must be valid JSON

---

Project structure

stubllm/
├── src/stubllm/
│   ├── fixtures/       # YAML/JSON loading, Pydantic models, matching engine
│   ├── providers/      # OpenAI, Anthropic, Gemini endpoint handlers
│   ├── streaming/      # SSE streaming simulation
│   ├── recorder/       # Record-and-replay proxy
│   ├── pytest_plugin/  # pytest fixtures and @use_fixtures decorator
│   ├── server.py       # FastAPI app factory
│   └── cli.py          # click CLI
├── tests/              # >80% coverage
└── examples/           # Working examples (basic + advanced)

---

Contributing

git clone https://github.com/your-org/stubllm
cd stubllm
pip install -e ".[dev]"
pytest tests/ -v

---

License

MIT