llmbroker
Turn a crowd of free, rate-limited LLMs into one reliable model — no premium subscription, no single point of failure. No LangChain, no heavy deps.
pip install llmbroker
llmbroker env freetier > .env # which API keys to get, and where
broker = llmbroker.Broker() # no config file: the curated pool of free models
reply = broker.ask("Explain decorators in one sentence")
print(reply.text) # groq rate-limited? gemini answers instead
Fill in whichever keys are easy — models without keys just stay inactive.
Why another router? LiteLLM or OpenRouter forward your request and hand back the error; llmbroker runs the pool for you: backs off on rate limits and retries with the next model inside the same call, disables dead keys on its own, and learns which models are weak at which tasks. Set it up once, never administer it.
| Fast, resilient answers | Automatic failover by default; fastest_of=2 races models when latency matters |
| Chat, tools & agents | broker.chat(messages, tools=...), run_tool_loop(...) |
| Async & streaming | AsyncBroker — same engine for FastAPI / agents / workers, token by token |
| A paid model by name | Broker(direct=["opus"]) — an eternal alias, called past the pool |
| Scale out | Broker("postgresql://…") — sqlite / Postgres / MongoDB, calling code unchanged |
| Self-regulating pool | reply.record_quality(0.3) — weak models sink per task kind; rate later with broker.record_quality(...) |
| Nothing hidden | broker.snapshot(), the call journal, tracing by your own trace_id |
| Pluggable secrets | env vars, DB, AWS, Vault, or your own backend |
| Multi-user mode | per-user API keys on top of one shared pool |
Development
Do not forget to run . ./activate.sh. It needs uv installed.
Use pre-commit hooks for code quality:
pre-commit install
Install invoke preferably with uv tool:
uv tool install invoke
For a list of available scripts run invoke --list; for details on one, invoke <script> --help.
The bundled freetier preset drifts as providers change their free tiers; refresh
it with invoke catalog-refresh, which prints the maintenance runbook
(src/llmbroker/presets/freetier-refresh-prompt.md).
Reports:
Created with cookiecutter using template
1 maintainer
Metadata
1 maintainer
Release files for llmbroker 1.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| llmbroker-1.8.0.tar.gz | 596.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| llmbroker-1.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 728.6 kB
Release files / llmbroker-1.8.0.tar.gz
| Download URL | llmbroker-1.8.0.tar.gz |
|---|---|
| Size | 596.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9587ba3e561e15cf1778b2150fbee4c5f91ff953c316cda587c9a1bfc3a26e59
|
|
BLAKE2b-256 checksum How to use checksums |
e94f305a4e004cb9e8ad8eb997bc4d945635f6864581ea29f3b05c474e3fe118
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / llmbroker-1.8.0-py3-none-any.whl
| Download URL | llmbroker-1.8.0-py3-none-any.whl |
|---|---|
| Size | 132.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
21a6f1b01b0a0a85630886897a0f2bc65542c500033b18a73eef922c65479ceb
|
|
BLAKE2b-256 checksum How to use checksums |
9ddb4870f6218961f589d6b722cac5994b32983d6a2a0fe01fe3d0df95ee9ce1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
