Skip to main content
cachemaxxing logo

cachemaxxing

PyPI Python CI License: MIT

Stop paying full price for the same tokens.

Anthropic, OpenAI and Gemini bill a repeated prompt prefix at a fraction of the input price, but only when the bytes really repeat. cachemaxxing finds where they don't, tells you what it costs, and lays your prompts out so they do. It is a thin library, not a proxy: zero runtime dependencies, and you keep your own SDK client.

  • Measure. usage() turns any provider's response into the same numbers: input, cache read, cache write, output, estimated $ and hit rate. cachemaxxing report totals them per call site.
  • Diagnose. Give Stats each request and it tells you, as soon as it happens, the first byte where a request stopped repeating the previous one, why, and how to fix it. cachemaxxing report prints the same from a log.
  • Fix. Prompt orders your prompt from most to least stable and places each provider's cache markers for you.
from openai import OpenAI
from cachemaxxing import Prompt, usage

chat = [
    {"role": "system", "content": SYSTEM},  # your long, fixed instructions: cached
    {"role": "user", "content": "How do I reset my router?"},
]
p = Prompt(call_site="support.answer").messages(chat)  # call_site: any name for this call, used in reports

resp = OpenAI().chat.completions.create(**p.openai_chat("gpt-6-luna"))
# same p: anthropic.messages.create(max_tokens=1024, **p.anthropic("claude-haiku-5-5"))
#         genai_client.models.generate_content(**p.gemini("gemini-3.8-flash"))

u = usage(resp)
print(f"{u.hit_rate:.0%} of input from cache, ~${u.est_cost or 0:.4f}")

The system prompt and the conversation so far get the provider's cache markers, so each turn reads everything before the new question from cache. For tools, RAG chunks or a timestamp, build the prompt from parts: p.tools(TOOLS).system(SYSTEM).history(chat).retrieved(chunks).user(question). The quickstart shows where the markers go, and the exact request.

Works with

How
Anthropic client.messages.create(**p.anthropic(model))
OpenAI client.responses.create(**p.openai(model)) or client.chat.completions.create(**p.openai_chat(model))
Gemini client.models.generate_content(**p.gemini(model)), with an explicit cache via gemini_cache() when should_cache() says it pays
LiteLLM litellm.completion(**p.litellm("anthropic/claude-haiku-5-5"))
LangChain agents create_agent(..., middleware=[CacheMiddleware("support.agent")])
LangChain messages, LangGraph graphs, LCEL llm = CachedModel(chat_model), then llm.invoke([SystemMessage(...), HumanMessage(...)])

On OpenAI GPT-5.6+ it places explicit cache breakpoints. On Anthropic it places cache_control markers, at most 4. Gemini stays on implicit caching unless an explicit cache pays.

Find the misses

$ cachemaxxing report calls.jsonl --min-hit-rate 0.6     # from record() logs; exits 1 below the threshold
call site   calls  hit rate  input   cache read  cache write  output  est $   top drift
rag.answer  4      88.8%     31,574  28,024      714          2,601   0.0020

$ cachemaxxing diff request_1.json request_2.json
diverges at system[0].text char 41 (timestamp), after ~14 matching tokens
  a: 'You are helpful. Today is 2026-10-30T10:00:00Z.'
  b: 'You are helpful. Today is 2026-10-30T10:05:00Z.'

Add --html report.html for the same report as a self-contained dashboard page. See the demo, built from a real log. For the provider's own verdict, add diagnose= to an adapter; usage(resp).miss_reason then says system_changed, tools_changed, and so on. To gate CI on hit rate, use the bundled GitHub Action: uses: ashhadahsan/cachemaxxing@v0.10.0 with log: and min-hit-rate:.

Before you put it in a production request path, read Running in production. It covers rollout, what happens when something fails (cachemaxxing never fails a call that would have worked without it), logs without prompt text, concurrency and the stability policy.

Benchmark

Cached input tokens per conversation: naive vs clean prompt with provider auto-caching vs cachemaxxing

  • Against a sloppy app (the naive arm, chat and RAG): input cost is 76–88% lower on every model.
  • Against a clean prompt using the provider's own one-line caching, it depends on the provider:
    • Anthropic: automatic caching puts its single cache entry after the per-turn context (time, request id, retrieved chunks). That entry changes every turn, so the app pays the 1.25× write premium on every call and reads nothing back. That is 0% cached, and dearer than not caching at all. A breakpoint after the static prompt, which cachemaxxing places, fixes it: 82–89% cheaper.
    • Gemini: implicit caching is best-effort and reached 39–56%. The explicit cache cachemaxxing creates when should_cache() says it pays reached 88–97%, 62–67% cheaper, with creation and storage included.
    • OpenAI: implicit caching already finds a stable prefix, and cachemaxxing is only 6–19% cheaper. On OpenAI the win is fixing the habits, plus the measuring tools.
    • LangChain agents: CacheMiddleware ties LangChain's AnthropicPromptCachingMiddleware on Anthropic and the default agent on OpenAI. It adds one setup for every provider and per-call-site measurement, not extra savings.
Workload Model naive clean + provider auto-caching cachemaxxing vs naive vs clean
chat claude-haiku-5-5 0% · $0.0090 0% · $0.0112 94.4% · $0.0014 84% cheaper 87% cheaper
chat gpt-6-luna 0% · $0.0068 92.8% · $0.0009 94% · $0.0009 87% cheaper 6% cheaper
chat gemini-3.8-flash 0% · $0.0468 55.6% · $0.0236 97.3% · $0.0077 83% cheaper 67% cheaper
chat claude-sonnet-5-5 0% · $0.1791 0% · $0.2232 92.4% · $0.0247 86% cheaper 89% cheaper
chat gpt-6.1-sol 0% · $0.1389 88.4% · $0.0209 91% · $0.0168 88% cheaper 19% cheaper
rag claude-haiku-5-5 0% · $0.0069 0% · $0.0086 87.4% · $0.0015 78% cheaper 82% cheaper
rag gpt-6-luna 0% · $0.0049 84.1% · $0.0011 85.6% · $0.0010 80% cheaper 15% cheaper
rag gemini-3.8-flash 0% · $0.0352 39.1% · $0.0226 88.2% · $0.0086 76% cheaper 62% cheaper
agent claude-haiku-5-5 0% · $0.0044 89.8% · $0.0010 89.9% · $0.0009 78% cheaper 1% cheaper
agent gpt-6-luna 90.4% · $0.0006 — 90.1% · $0.0006 2% dearer —

Cells: cached share of input tokens · estimated input cost per conversation at list price (cache writes and explicit-cache storage included; output excluded, since answer length varies between runs regardless of caching).

Setup: 8-turn conversations (agents: 3 questions with a tool call), 3 customers per arm. Every arm starts from a cold cache with its own prompt, so no arm reads another's entries. The naive arm has the four habits: a timestamp and request id at the top of the system prompt, tool order that varies, JSON key order that varies, and history re-rendered every turn; in RAG, chunks also go in the system prompt. The clean arm has the same content laid out well, with Anthropic's top-level cache_control or OpenAI's and Gemini's default implicit caching. Data: benchmarks/results/, run 2026-10-30. Rerun with uv run --env-file .env python benchmarks/bench_<provider>.py --workload chat|rag and benchmarks/bench_agent.py; the table comes from benchmarks/summary.py.

Examples

Install

pip install cachemaxxing                  # or: uv add cachemaxxing
pip install "cachemaxxing[anthropic]"     # extras: anthropic, openai, gemini, litellm, langchain

New here? The tutorial takes ten minutes, from a broken cache to a fixed one.

Docs: https://ashhadahsan.github.io/cachemaxxing/ covers the tutorial, the quickstart, how caching works per provider, diagnosing a low hit rate, LangChain and LangGraph, the API reference, and how it compares.

MIT licensed.

Metadata

Release files for cachemaxxing 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cachemaxxing 0.10.0
File Size Uploaded Provenance Actions
cachemaxxing-0.10.0.tar.gz 41.1 kB With provenance Details for cachemaxxing-0.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cachemaxxing 0.10.0
File Interpreter ABI Platform Actions
cachemaxxing-0.10.0-py3-none-any.whl Python 3 none any Details for cachemaxxing-0.10.0-py3-none-any.whl

Total release size: 87.0 kB

Release files / cachemaxxing-0.10.0.tar.gz

Download URL cachemaxxing-0.10.0.tar.gz
Size 41.1 kB
Tags Source
SHA-256 checksum
How to use checksums
4a7d0b38f0dee2a4d80d2cff9e64fc23156541d332dfc142a0588db30db1acca
BLAKE2b-256 checksum
How to use checksums
cffdd710ab1d1ac003c10c86b1a8c1ef4ca4e8b0adcc024b5c9cb0985d48d1be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 31, 2026.

Transparency log

Release files / cachemaxxing-0.10.0-py3-none-any.whl

Download URL cachemaxxing-0.10.0-py3-none-any.whl
Size 45.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9afd38a03af1a13d30158de40ec43e814c7848d4afaa84ec37bfbc7b7b3317c9
BLAKE2b-256 checksum
How to use checksums
2867b51aeacb6935f3240ee6e95f09cfa36b028716d7e8392e5c3b2c6b2cbe05
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 31, 2026.

Transparency log

Security for cachemaxxing 0.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for cachemaxxing-0.10.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 31, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page