llmtest
The pytest for LLMs. Test your AI outputs like you test your code.
Quick Start
pip install pytest-llmtest[anthropic]
from llmtest import expect, llm_test
@llm_test(
expect.contains("Paris"),
expect.latency_under(2000),
expect.cost_under(0.001),
model="claude-sonnet-4-20250514",
)
def test_capital(llm):
output = llm("What is the capital of France?")
assert "Paris" in output.content
pytest test_capitals.py -v
Features
- 22+ assertions — text, performance, agent, composable
- Zero LLM calls for most checks — deterministic and instant
- Built on Pydantic — auto-validation, JSON serialization, schema generation
- Multi-provider — OpenAI, Anthropic, Ollama out of the box
- Agent testing — tool calls, loop detection, call ordering
- Retry support — handle non-deterministic outputs
- CI/CD native — pytest markers, JUnit XML, JSON reporters
- Pydantic AI integration — test your existing agents
Installation
pip install pytest-llmtest[anthropic] # Anthropic
pip install pytest-llmtest[openai] # OpenAI
pip install pytest-llmtest[ollama] # Ollama (local)
pip install pytest-llmtest[all] # Everything
Documentation
Full docs at docs.llmtest.dev
Contributing
git clone https://github.com/bahadiraraz/llmtest
cd llmtest
uv sync --all-extras
uv run pytest tests/ -v
License
MIT
Metadata
Release files for pytest-llmtest 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pytest_llmtest-0.1.0.tar.gz | 4.6 kB | Without provenance | Details for pytest_llmtest-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pytest_llmtest-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for pytest_llmtest-0.1.0-py3-none-any.whl |
Total release size: 8.6 kB
Release files / pytest_llmtest-0.1.0.tar.gz
| Download URL | pytest_llmtest-0.1.0.tar.gz |
|---|---|
| Size | 4.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
293d0229af52ed3ef43701067b83b2e14d8bde241b7c681b51342045f909c9ce
|
|
BLAKE2b-256 checksum How to use checksums |
498fa622ea9650e3bd614dd625e88b8f1055b34db57f22976f49bb6808a07d5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
Release files / pytest_llmtest-0.1.0-py3-none-any.whl
| Download URL | pytest_llmtest-0.1.0-py3-none-any.whl |
|---|---|
| Size | 4.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d876ad8a8b9d358a5f63deb7af27588a1018925ccb50c05c8298a93f5468545b
|
|
BLAKE2b-256 checksum How to use checksums |
802aa207dd0c2de49e4dbbe713a9512f370922cb540b4c2b6653cc86cb3be584
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.0
|
Security for pytest-llmtest 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.