pytest-llmtest 0.1.0

The pytest for LLMs — fast, Pydantic-based assertions for AI applications

llmtest

The pytest for LLMs. Test your AI outputs like you test your code.

Quick Start

pip install pytest-llmtest[anthropic]

from llmtest import expect, llm_test

@llm_test(
    expect.contains("Paris"),
    expect.latency_under(2000),
    expect.cost_under(0.001),
    model="claude-sonnet-4-20250514",
)
def test_capital(llm):
    output = llm("What is the capital of France?")
    assert "Paris" in output.content

pytest test_capitals.py -v

Features

• 22+ assertions — text, performance, agent, composable
• Zero LLM calls for most checks — deterministic and instant
• Built on Pydantic — auto-validation, JSON serialization, schema generation
• Multi-provider — OpenAI, Anthropic, Ollama out of the box
• Agent testing — tool calls, loop detection, call ordering
• Retry support — handle non-deterministic outputs
• CI/CD native — pytest markers, JUnit XML, JSON reporters
• Pydantic AI integration — test your existing agents

Installation

pip install pytest-llmtest[anthropic]    # Anthropic
pip install pytest-llmtest[openai]       # OpenAI
pip install pytest-llmtest[ollama]       # Ollama (local)
pip install pytest-llmtest[all]          # Everything

Documentation

Full docs at docs.llmtest.dev

Contributing

git clone https://github.com/bahadiraraz/llmtest
cd llmtest
uv sync --all-extras
uv run pytest tests/ -v

License

MIT