llmbroker
Turn a crowd of free, rate-limited LLMs into one reliable model — no premium subscription, no single point of failure. No LangChain, no heavy deps.
pip install llmbroker
llmbroker env freetier > .env # which API keys to get, and where
broker = llmbroker.Broker() # no config file: the curated pool of free models
reply = broker.ask("Explain decorators in one sentence")
print(reply.text) # groq rate-limited? gemini answers instead
Fill in whichever keys are easy — models without keys just stay inactive.
Why another router? LiteLLM or OpenRouter forward your request and hand back the error; llmbroker runs the pool for you: backs off on rate limits and retries with the next model inside the same call, disables dead keys on its own, and learns which models are weak at which tasks. Set it up once, never administer it.
| Fast, resilient answers | Automatic failover by default; fastest_of=2 races models when latency matters |
| Chat, tools & agents | broker.chat(messages, tools=...), run_tool_loop(...) |
| Async & streaming | AsyncBroker — same engine for FastAPI / agents / workers, token by token |
| A paid model by name | Broker(direct=["opus"]) — an eternal alias, called past the pool |
| Scale out | Broker("postgresql://…") — sqlite / Postgres / MongoDB, calling code unchanged |
| Self-regulating pool | reply.record_quality(0.3) — weak models sink per task kind; rate later with broker.record_quality(...) |
| Nothing hidden | broker.snapshot(), the call journal, tracing by your own trace_id |
| Pluggable secrets | env vars, DB, AWS, Vault, or your own backend |
| Multi-user mode | per-user API keys on top of one shared pool |
Development
Do not forget to run . ./activate.sh. It needs uv installed.
Use pre-commit hooks for code quality:
pre-commit install
Install invoke preferably with uv tool:
uv tool install invoke
For a list of available scripts run invoke --list; for details on one, invoke <script> --help.
The bundled freetier preset drifts as providers change their free tiers; refresh
it with invoke catalog-refresh, which prints the maintenance runbook
(src/llmbroker/presets/freetier-refresh-prompt.md).
Reports:
Created with cookiecutter using template
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file llmbroker-1.7.0.tar.gz.
File metadata
- Download URL: llmbroker-1.7.0.tar.gz
- Upload date:
- Size: 572.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
641b7a7b6b0a85264a501e4f12ae2ddb88cd7c60dc9d095a24680fccaad9ce81
|
|
| MD5 |
74b34f74c8c95b445ed73f62f4d326af
|
|
| BLAKE2b-256 |
9427f323017b01605e155389fc65327663fd4b36f70cd92236e7f7b7f1527a95
|
File details
Details for the file llmbroker-1.7.0-py3-none-any.whl.
File metadata
- Download URL: llmbroker-1.7.0-py3-none-any.whl
- Upload date:
- Size: 128.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e0319da6fc399499dcf4c71e072266402df64d0c15012c60e7b1371d666a42d6
|
|
| MD5 |
40876cbb3a3212998716c808adc42689
|
|
| BLAKE2b-256 |
7213a30a35e779fab7b293d206dc0530c6c9c097cf9c7973fcd2a5b009a258ec
|
