llmtest
The pytest for LLMs. Test your AI outputs like you test your code.
Quick Start
pip install pytest-llmtest[anthropic]
from llmtest import expect, llm_test
@llm_test(
expect.contains("Paris"),
expect.latency_under(2000),
expect.cost_under(0.001),
model="claude-sonnet-4-20250514",
)
def test_capital(llm):
output = llm("What is the capital of France?")
assert "Paris" in output.content
pytest test_capitals.py -v
Features
- 22+ assertions — text, performance, agent, composable
- Zero LLM calls for most checks — deterministic and instant
- Built on Pydantic — auto-validation, JSON serialization, schema generation
- Multi-provider — OpenAI, Anthropic, Ollama out of the box
- Agent testing — tool calls, loop detection, call ordering
- Retry support — handle non-deterministic outputs
- CI/CD native — pytest markers, JUnit XML, JSON reporters
- Pydantic AI integration — test your existing agents
Installation
pip install pytest-llmtest[anthropic] # Anthropic
pip install pytest-llmtest[openai] # OpenAI
pip install pytest-llmtest[ollama] # Ollama (local)
pip install pytest-llmtest[all] # Everything
Documentation
Full docs at docs.llmtest.dev
Contributing
git clone https://github.com/bahadiraraz/llmtest
cd llmtest
uv sync --all-extras
uv run pytest tests/ -v
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for pytest-llmtest 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pytest_llmtest-0.1.0.tar.gz | 4.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pytest_llmtest-0.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 8.6 kB
Release files / pytest_llmtest-0.1.0.tar.gz
| Download URL | pytest_llmtest-0.1.0.tar.gz |
|---|---|
| Size | 4.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
293d0229af52ed3ef43701067b83b2e14d8bde241b7c681b51342045f909c9ce
|
|
BLAKE2b-256 checksum How to use checksums |
498fa622ea9650e3bd614dd625e88b8f1055b34db57f22976f49bb6808a07d5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
Release files / pytest_llmtest-0.1.0-py3-none-any.whl
| Download URL | pytest_llmtest-0.1.0-py3-none-any.whl |
|---|---|
| Size | 4.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d876ad8a8b9d358a5f63deb7af27588a1018925ccb50c05c8298a93f5468545b
|
|
BLAKE2b-256 checksum How to use checksums |
802aa207dd0c2de49e4dbbe713a9512f370922cb540b4c2b6653cc86cb3be584
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
