Skip to main content

pytest-llmchaos

Give your LLM calls a bad day before production does.

pytest-llmchaos is a pytest plugin for code that calls the OpenAI or Anthropic Python SDKs. It records real responses once, replays them offline in every later run, and lets each test inject the failures that break LLM features in production:

  • rate limits and overloaded servers
  • timeouts and dropped connections
  • streams that die halfway
  • answers cut off at the token limit
  • tool calls with broken arguments

The faults look exactly like the real thing to the SDKs, so they raise their real exception classes and run their real retry logic. Your error handling is tested the way it runs in production, with no network, no API key and no cost.

Install

pip install pytest-llmchaos

The plugin activates automatically. Use the llm fixture in any test that calls a provider.

Example

import pytest
from llmchaos import faults


def test_summarize(llm):
    assert "refund" in summarize(ticket)  # replayed from cassettes/


def test_survives_rate_limits(llm):
    llm.inject(faults.rate_limit(times=2))  # two real-looking 429s, then the recorded answer
    assert summarize(ticket)
    assert llm.attempts == 3  # proves the SDK retried


def test_falls_back_when_openai_keeps_failing(llm):
    llm.inject(faults.overloaded(times=3))  # more than the SDK's 2 default retries
    assert summarize(ticket) == FALLBACK_SUMMARY


def test_handles_a_cut_stream(llm):
    llm.inject(faults.stream_cut(after_events=4))
    with pytest.raises(PartialResponseError):  # your app's own error
        stream_summary(ticket)

The first run with an API key records real responses to cassettes/<module>/<test>.json next to the test file. Every later run replays them. examples/support_bot is a complete, runnable example.

Faults

Fault What the SDK sees
rate_limit(times=1, retry_after=0.001) HTTP 429 with the provider's error body and retry-after headers
overloaded(times=1) Anthropic HTTP 529 overloaded_error; OpenAI HTTP 503
server_error(times=1) HTTP 500
timeout(times=1) ReadTimeout from the HTTP library
connection_error(times=1) ConnectError from the HTTP library
stream_cut(after_events=1) The recorded stream stops after N events with RemoteProtocolError
truncated() The first half of the text, ending with finish_reason="length" / stop_reason="max_tokens"
bad_tool_args() OpenAI: tool arguments that are not valid JSON. Anthropic: a tool input with a key missing

Every fault also takes on_call=n to target the nth call in the test. By default it targets the next call.

  • Replace faults (the first five) answer an HTTP attempt themselves and don't use up the recording. rate_limit(times=2) means two 429s, then the real answer on the SDK's third attempt. Use a times larger than the SDK's retry budget to test the "gives up" path.
  • Mutate faults (the last three) alter the recorded response of the call they target.
  • Strict by default. A test fails if a fault it injected never fired, because that means the code never made the call you thought it did.
  • Fast by default. Synthesized HTTP errors carry retry-after-ms: 1, so SDK retries are instant. timeout and connection_error have no response to carry that header, so retries after them use the SDK's own backoff (about 0.5–1 s each).
  • Synthesized error messages end with (simulated by llmchaos), so they can't be mistaken for real outages in logs.

Recording

--llm-record= Cassette exists Cassette missing
none replay fail with CassetteMissingError
missing (default) replay record
all re-record record

Use none in CI, so an unrecorded call fails loudly instead of reaching the network:

pytest --llm-record=none

A request matches a recording when its method, URL and JSON body are equal; key order doesn't matter. On a mismatch, the test fails with a diff of what changed. To exclude dynamic values, list dotted paths. * matches every key or index:

[pytest]
llmchaos_ignore =
    messages.*.metadata
    user

To exclude a path for a single test, call llm.ignore("messages.0.content").

Other options:

  • llmchaos_cassette_dir (default cassettes)
  • llmchaos_openai_hosts and llmchaos_anthropic_hosts, for gateways and Azure endpoints

Secrets

Cassettes are safe to commit.

  • Headers come from an allowlist: content-type, anthropic-version, anthropic-beta, openai-beta, and response request IDs. authorization and x-api-key are never written.
  • Every string is scanned before saving, for sk-… keys and for the values of OPENAI_API_KEY / ANTHROPIC_API_KEY. If anything matches, the write is refused with SecretLeakError.

Supported versions

  • Python 3.11–3.14
  • openai 3.x and anthropic 1.x/2.x (built on httpx2), tested on their latest releases
  • openai 2.x and anthropic 0.x (built on httpx), tested on openai==2.46.0 and anthropic==0.125.0

Limitations

  • Only tests that use the llm fixture are intercepted.
  • Apps that swap in a non-httpx HTTP backend (such as the OpenAI SDK's optional aiohttp client) are not intercepted.
  • truncated and bad_tool_args support streaming for Chat Completions and Anthropic Messages, but only non-streaming bodies for the OpenAI Responses API.
  • Recording reads each stream fully before replaying it to the SDK, so real-time chunk timing is not preserved.
  • With concurrent async calls, on_call counts calls in arrival order.

Contributing

See CONTRIBUTING.md.

License

MIT

Metadata

Release files for pytest-llmchaos 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-llmchaos 0.1.0
File Size Uploaded Provenance Actions
pytest_llmchaos-0.1.0.tar.gz 113.1 kB With provenance Details for pytest_llmchaos-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-llmchaos 0.1.0
File Interpreter ABI Platform Actions
pytest_llmchaos-0.1.0-py3-none-any.whl Python 3 none any Details for pytest_llmchaos-0.1.0-py3-none-any.whl

Total release size: 137.1 kB

Release files / pytest_llmchaos-0.1.0.tar.gz

Download URL pytest_llmchaos-0.1.0.tar.gz
Size 113.1 kB
Tags Source
SHA-256 checksum
How to use checksums
49c2a39edab9deeda7ffe39ad1bd0d6ddb9c29a406c530409b50efbe5a47b9dc
BLAKE2b-256 checksum
How to use checksums
fdd1096c466183cd29e27bfafadd8adef4c68ebeb21f8fbd40df41ba96edacfa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / pytest_llmchaos-0.1.0-py3-none-any.whl

Download URL pytest_llmchaos-0.1.0-py3-none-any.whl
Size 24.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
295bb6612e83cfdf3af032d42a0b90a6681d2a31332759927ce984f652d65ff2
BLAKE2b-256 checksum
How to use checksums
ecbc82040ea9fbf8058d8bf8ecdde409aa207e9b555c35e6e1f7a84618697567
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for pytest-llmchaos 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pytest_llmchaos-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page