llm-governor

Client-side rate limiting for the Claude API. Stop before the server has to say no.

This library never sees your API key. It wraps calls you make with your own client, and reads only anthropic-ratelimit-* response headers. No credentials are required, requested, logged, or stored.

pip install llm-governor

Why

Claude's Messages API enforces three independent limits per model: requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM). Blow any one of them and you get a 429.

Most retry-on-429 code reacts after the failure. llm-governor mirrors the server's own token bucket locally, corrects itself from the rate-limit headers on every response, and paces your requests so the 429 never happens.

It also understands cached input. On most Claude models, cache_read_input_tokens do not count toward ITPM. A naive limiter that counts every input token will throttle you to a fraction of your real throughput; this one tracks your rolling cache-hit rate and reserves only what actually counts.

Quick start

import anthropic
from llm_governor import Governor

client = anthropic.Anthropic()          # your key, your client
gov = Governor(model="claude-sonnet-5") # knows nothing about keys

with gov.slot(messages=messages, max_tokens=1024) as slot:
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=messages,
    )
    slot.observe(resp)   # feeds headers + usage back in

The first call runs against conservative defaults. From the second call onward, the governor knows your real limits because the API told it.

Async

gov = Governor(model="claude-sonnet-5", max_concurrency=8)

async def worker(messages):
    async with gov.aslot(messages=messages, max_tokens=1024) as slot:
        resp = await client.messages.create(...)
        slot.observe(resp)
        return resp

await asyncio.gather(*(worker(m) for m in batch))

Live dashboard

pip install "llm-governor[dashboard]"
llm-governor demo          # simulated workload, no API key needed

from llm_governor.dashboard import Dashboard

with Dashboard(gov):
    run_my_batch()

╭─ llm-governor · claude-sonnet-5 ─────────────────────╮
│  RPM   ████████████████░░░░   812 / 1000             │
│  ITPM  ███████░░░░░░░░░░░░░   1.4M / 2.0M            │
│  OTPM  ██████████████████░░   361K / 400K            │
│                                                      │
│  cache  78.0%                        94 req/min      │
│  flight     7                        231 done        │
│  429s       0                        3.2s waited     │
╰────────────────────────────── live ──────────────────╯

Bars turn amber below 25% headroom and red below 10%.

Dry run

See whether a workload would throttle, without waiting for it:

gov = Governor(model="claude-sonnet-5", dry_run=True)
# ... replay your workload ...
print(gov.snapshot()["throttled_seconds"])

Options

On credentials

The header allowlist in llm_governor.headers.ALLOWED_HEADERS is explicit and tested. scrub() is the only path by which headers enter this library, and a test asserts that x-api-key and authorization never survive it. If you extend this library, do not replace that with dict(response.headers).

Notes and limits

• State is per-process. For multiple workers sharing one org limit, divide headroom by your worker count, or wait for the planned Redis backend.
• Input estimation is character-based and approximate. Pass estimated_input explicitly if you have a real count from the token-counting endpoint.
• Batch API requests have separate limits and are not covered.
• max_tokens is reserved pessimistically and refunded once the real output size is known.

License

MIT