llmtest
The pytest for LLMs. Test your AI outputs like you test your code.
Quick Start
pip install pytest-llmtest[anthropic]
from llmtest import expect, llm_test
@llm_test(
expect.contains("Paris"),
expect.latency_under(2000),
expect.cost_under(0.001),
model="claude-sonnet-4-20250514",
)
def test_capital(llm):
output = llm("What is the capital of France?")
assert "Paris" in output.content
pytest test_capitals.py -v
Features
- 22+ assertions — text, performance, agent, composable
- Zero LLM calls for most checks — deterministic and instant
- Built on Pydantic — auto-validation, JSON serialization, schema generation
- Multi-provider — OpenAI, Anthropic, Ollama out of the box
- Agent testing — tool calls, loop detection, call ordering
- Retry support — handle non-deterministic outputs
- CI/CD native — pytest markers, JUnit XML, JSON reporters
- Pydantic AI integration — test your existing agents
Installation
pip install pytest-llmtest[anthropic] # Anthropic
pip install pytest-llmtest[openai] # OpenAI
pip install pytest-llmtest[ollama] # Ollama (local)
pip install pytest-llmtest[all] # Everything
Documentation
Full docs at docs.llmtest.dev
Contributing
git clone https://github.com/bahadiraraz/llmtest
cd llmtest
uv sync --all-extras
uv run pytest tests/ -v
License
MIT
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pytest_llmtest-0.1.0.tar.gz
(4.6 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pytest_llmtest-0.1.0.tar.gz.
File metadata
- Download URL: pytest_llmtest-0.1.0.tar.gz
- Upload date:
- Size: 4.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.14.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
293d0229af52ed3ef43701067b83b2e14d8bde241b7c681b51342045f909c9ce
|
|
| MD5 |
5014ff8671f883e47c6a14a352fbc9ec
|
|
| BLAKE2b-256 |
498fa622ea9650e3bd614dd625e88b8f1055b34db57f22976f49bb6808a07d5c
|
File details
Details for the file pytest_llmtest-0.1.0-py3-none-any.whl.
File metadata
- Download URL: pytest_llmtest-0.1.0-py3-none-any.whl
- Upload date:
- Size: 4.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.14.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d876ad8a8b9d358a5f63deb7af27588a1018925ccb50c05c8298a93f5468545b
|
|
| MD5 |
14d4e11cbc7b9023351ce25b36f1114f
|
|
| BLAKE2b-256 |
802aa207dd0c2de49e4dbbe713a9512f370922cb540b4c2b6653cc86cb3be584
|
