pytest-llmchaos
Give your LLM calls a bad day before production does.
pytest-llmchaos is a pytest plugin for code that calls the OpenAI or Anthropic Python SDKs. It records real responses once, replays them offline in every later run, and lets each test inject the failures that break LLM features in production:
- rate limits and overloaded servers
- timeouts and dropped connections
- streams that die halfway
- answers cut off at the token limit
- tool calls with broken arguments
The faults look exactly like the real thing to the SDKs, so they raise their real exception classes and run their real retry logic. Your error handling is tested the way it runs in production, with no network, no API key and no cost.
Install
pip install pytest-llmchaos
The plugin activates automatically. Use the llm fixture in any test that calls a provider.
Example
import pytest
from llmchaos import faults
def test_summarize(llm):
assert "refund" in summarize(ticket) # replayed from cassettes/
def test_survives_rate_limits(llm):
llm.inject(faults.rate_limit(times=2)) # two real-looking 429s, then the recorded answer
assert summarize(ticket)
assert llm.attempts == 3 # proves the SDK retried
def test_falls_back_when_openai_keeps_failing(llm):
llm.inject(faults.overloaded(times=3)) # more than the SDK's 2 default retries
assert summarize(ticket) == FALLBACK_SUMMARY
def test_handles_a_cut_stream(llm):
llm.inject(faults.stream_cut(after_events=4))
with pytest.raises(PartialResponseError): # your app's own error
stream_summary(ticket)
The first run with an API key records real responses to cassettes/<module>/<test>.json next to the test file. Every later run replays them. examples/support_bot is a complete, runnable example.
Faults
| Fault | What the SDK sees |
|---|---|
rate_limit(times=1, retry_after=0.001) |
HTTP 429 with the provider's error body and retry-after headers |
overloaded(times=1) |
Anthropic HTTP 529 overloaded_error; OpenAI HTTP 503 |
server_error(times=1) |
HTTP 500 |
timeout(times=1) |
ReadTimeout from the HTTP library |
connection_error(times=1) |
ConnectError from the HTTP library |
stream_cut(after_events=1) |
The recorded stream stops after N events with RemoteProtocolError |
truncated() |
The first half of the text, ending with finish_reason="length" / stop_reason="max_tokens" |
bad_tool_args() |
OpenAI: tool arguments that are not valid JSON. Anthropic: a tool input with a key missing |
Every fault also takes on_call=n to target the nth call in the test. By default it targets the next call.
- Replace faults (the first five) answer an HTTP attempt themselves and don't use up the recording.
rate_limit(times=2)means two 429s, then the real answer on the SDK's third attempt. Use atimeslarger than the SDK's retry budget to test the "gives up" path. - Mutate faults (the last three) alter the recorded response of the call they target.
- Strict by default. A test fails if a fault it injected never fired, because that means the code never made the call you thought it did.
- Fast by default. Synthesized HTTP errors carry
retry-after-ms: 1, so SDK retries are instant.timeoutandconnection_errorhave no response to carry that header, so retries after them use the SDK's own backoff (about 0.5–1 s each). - Synthesized error messages end with
(simulated by llmchaos), so they can't be mistaken for real outages in logs.
Recording
--llm-record= |
Cassette exists | Cassette missing |
|---|---|---|
none |
replay | fail with CassetteMissingError |
missing (default) |
replay | record |
all |
re-record | record |
Use none in CI, so an unrecorded call fails loudly instead of reaching the network:
pytest --llm-record=none
A request matches a recording when its method, URL and JSON body are equal; key order doesn't matter. On a mismatch, the test fails with a diff of what changed. To exclude dynamic values, list dotted paths. * matches every key or index:
[pytest]
llmchaos_ignore =
messages.*.metadata
user
To exclude a path for a single test, call llm.ignore("messages.0.content").
Other options:
llmchaos_cassette_dir(defaultcassettes)llmchaos_openai_hostsandllmchaos_anthropic_hosts, for gateways and Azure endpoints
Secrets
Cassettes are safe to commit.
- Headers come from an allowlist:
content-type,anthropic-version,anthropic-beta,openai-beta, and response request IDs.authorizationandx-api-keyare never written. - Every string is scanned before saving, for
sk-…keys and for the values ofOPENAI_API_KEY/ANTHROPIC_API_KEY. If anything matches, the write is refused withSecretLeakError.
Supported versions
- Python 3.11–3.14
openai3.x andanthropic1.x/2.x (built onhttpx2), tested on their latest releasesopenai2.x andanthropic0.x (built onhttpx), tested onopenai==2.46.0andanthropic==0.125.0
Limitations
- Only tests that use the
llmfixture are intercepted. - Apps that swap in a non-httpx HTTP backend (such as the OpenAI SDK's optional aiohttp client) are not intercepted.
truncatedandbad_tool_argssupport streaming for Chat Completions and Anthropic Messages, but only non-streaming bodies for the OpenAI Responses API.- Recording reads each stream fully before replaying it to the SDK, so real-time chunk timing is not preserved.
- With concurrent async calls,
on_callcounts calls in arrival order.
Contributing
See CONTRIBUTING.md.
License
MIT
Metadata
Release files for pytest-llmchaos 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pytest_llmchaos-0.1.0.tar.gz | 113.1 kB | With provenance | Details for pytest_llmchaos-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pytest_llmchaos-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for pytest_llmchaos-0.1.0-py3-none-any.whl |
Total release size: 137.1 kB
Release files / pytest_llmchaos-0.1.0.tar.gz
| Download URL | pytest_llmchaos-0.1.0.tar.gz |
|---|---|
| Size | 113.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49c2a39edab9deeda7ffe39ad1bd0d6ddb9c29a406c530409b50efbe5a47b9dc
|
|
BLAKE2b-256 checksum How to use checksums |
fdd1096c466183cd29e27bfafadd8adef4c68ebeb21f8fbd40df41ba96edacfa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / pytest_llmchaos-0.1.0-py3-none-any.whl
| Download URL | pytest_llmchaos-0.1.0-py3-none-any.whl |
|---|---|
| Size | 24.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
295bb6612e83cfdf3af032d42a0b90a6681d2a31332759927ce984f652d65ff2
|
|
BLAKE2b-256 checksum How to use checksums |
ecbc82040ea9fbf8058d8bf8ecdde409aa207e9b555c35e6e1f7a84618697567
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logSecurity for pytest-llmchaos 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for pytest_llmchaos-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.