pytest-llm-cost
A pytest plugin that tells you what the LLM calls in your test suite cost, test by test, in US dollars. It can also fail the run when the suite goes over a budget.
$ pip install pytest-llm-cost
$ pytest --llm-budget 0.50
...
================================== LLM cost ===================================
test calls in out USD
tests/test_agent.py::test_plans_a_trip 6 48,200 3,900 $0.203100
tests/test_rag.py::test_answers_with_sources 2 12,400 610 $0.002226
tests/test_rag.py::test_refuses_off_topic 1 1,900 40 $0.000309
by model: anthropic/claude-sonnet-4-6 6 x $0.203100; openai/gpt-4o-mini 3 x $0.002535
total $0.205635 over 9 calls in 3 tests (llm-prices 1.3.0, checked 2026-11-15)
budget $0.5, 41% used
You do not need to change any code. Once installed, the plugin sees every response from:
- the OpenAI Python SDK: chat completions, the Responses API, embeddings,
parse, streaming, sync and async; - the Anthropic Python SDK: messages,
messages.stream(),create(stream=True), sync and async; - LiteLLM:
litellm.completionandlitellm.acompletion. Each call is counted once, even though LiteLLM calls the OpenAI SDK underneath.
It reads the token counts each response reports. That includes cache reads and writes, so prompt caching is priced at the cached rate. It then prices those counts with llm-prices, a dated table of list prices for OpenAI, Anthropic, Google, Bedrock, Vertex, Azure, DeepSeek, Mistral, Groq, Together, Fireworks, xAI, Cohere and OpenRouter. Run pip install -U llm-prices to get new prices without upgrading the plugin.
Options
| Option | ini key | What it does |
|---|---|---|
--llm-budget USD |
llm_budget |
Fails the run (exit code 1) when the LLM spend is over USD. |
--llm-cost-top N |
llm_cost_top |
Sets how many of the most expensive tests the report lists (default 10, 0 for all). |
--llm-cost-json PATH |
Writes every call, with its tokens, its price and the price row used, to a JSON file for CI dashboards. |
# pytest.ini
[pytest]
llm_budget = 2.00
The report only appears when the run made LLM calls or a budget is set. Turn the plugin off for one run with pytest -p no:llm_cost.
Budget per test
import pytest
@pytest.mark.llm_budget(0.02)
def test_summary_stays_cheap(agent):
agent.summarize(long_document)
The test fails when its own calls cost more than $0.02.
The llm_cost fixture
def test_cache_hits_make_the_second_call_cheaper(client, llm_cost):
with llm_cost.measure() as first:
client.ask(question)
with llm_cost.measure() as second:
client.ask(question)
assert second.total < first.total / 2
| Attribute | |
|---|---|
llm_cost.total |
USD spent by this test so far (a float; total_decimal is exact) |
llm_cost.calls |
every response seen: model, provider, input_tokens, output_tokens, usd, priced_as |
llm_cost.input_tokens, llm_cost.output_tokens |
token totals for the test |
llm_cost.unpriced |
calls with no price (unknown model, or a stream that reported no usage) |
llm_cost.measure() |
a context manager with the same attributes for only the calls made inside the block |
llm_cost.record(model, usage=..., provider=...) |
adds a call the plugin cannot see, such as another SDK, raw HTTP or a cassette |
llm_cost.session_total |
USD spent by the whole run so far |
How a call is priced
- The provider comes from the client's base URL:
api.openai.comis OpenAI,api.groq.comis Groq,*.openai.azure.comis Azure, and so on. For LiteLLM it comes from LiteLLM's own provider name. A request served by Groq is never priced at OpenAI's rate. - Any other host, such as a local server, a proxy or a mock, is priced by model id alone. A test that talks to a mocked
gpt-4otherefore shows what that call would cost against the real API. - A model llm-prices does not list (
llama3on Ollama, a fine-tune id) is counted and reported as unpriced, never guessed. - OpenAI streams only report usage when you pass
stream_options={"include_usage": True}. Streams without it are counted and listed under "no usage".
pytest-xdist
Each worker records its own calls and sends them to the controller, which prints one report and applies the budget to the whole run. Inside a test, llm_cost.session_total covers only that worker.
Notes
- The plugin hooks an SDK only once something imports it, so a run that never imports
litellmdoes not pay for importing it. - Prices are list prices. Batch discounts, regional surcharges and negotiated rates are not applied.
- A failure inside the plugin never breaks your API call. It emits one warning and leaves that call unrecorded.
MIT licensed. Maintained by Matthew Busel. Price corrections go to llm-prices.
Metadata
Release files for pytest-llm-cost 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pytest_llm_cost-0.1.0.tar.gz | 21.6 kB | With provenance | Details for pytest_llm_cost-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pytest_llm_cost-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for pytest_llm_cost-0.1.0-py3-none-any.whl |
Total release size: 38.8 kB
Release files / pytest_llm_cost-0.1.0.tar.gz
| Download URL | pytest_llm_cost-0.1.0.tar.gz |
|---|---|
| Size | 21.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bc439786b413bd9984be2341fd45bedd098c3305f9e7159da3eb1403339a2892
|
|
BLAKE2b-256 checksum How to use checksums |
bb43eb2f2388281742294b6b085df2db25e3a86647accb9c09ddc7ded8efcd4f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pytest_llm_cost-0.1.0-py3-none-any.whl
| Download URL | pytest_llm_cost-0.1.0-py3-none-any.whl |
|---|---|
| Size | 17.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a397140ab3c400de806880cf7e465aa396d7323ef7b7b0f8aa2688a2be44a5a3
|
|
BLAKE2b-256 checksum How to use checksums |
f6a7c80515d94b1ea6544e2a82298707183f0c6fc6fb9519a96f3551e9924b10
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for pytest-llm-cost 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for pytest_llm_cost-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.