llmbroker 1.1.0

No project description provided

llmbroker

Turn a crowd of free, rate-limited LLMs into one reliable model — no premium subscription, no single point of failure.

Many free LLMs are unreliable and mediocre on their own. llmbroker pools them and routes across the pool, turning quantity into dependable, good-enough quality — without paying for a premium model.

No LangChain, no heavy deps.

llms = llmbroker.Broker("llms.toml")
print(llms.ask("Explain Python decorators in one sentence").text)

Put your LLMs in llms.toml — or grab a preset:

llmbroker preset freetier > llms.toml

The only setup is the API keys. Ask llmbroker which ones your pool needs — each with a note on where to get it, ordered easiest-and-most-valuable first:

llmbroker env llms.toml > .env

Running with only some of the keys is the normal, intended mode — add the keys that are easy and worth it, and the pool assembles itself from whatever is present. A missing key just means that one model stays inactive; it is not an error.

The same Broker scales straight to a server or cluster — add Redis or Postgres to share cooldown state across instances, the calling code stays the same.

Why llmbroker:

• Automatic failover — when one LLM is rate-limited or down, the next one in the pool answers instead; you get a reply, not an error, as long as any LLM is up.
• Chat, tools & agents — one-shot ask, multi-turn chat, and function/tool calling for agentic workflows.
• Async-first — llmbroker.AsyncBroker built on asyncio for FastAPI, agents, and async workers; llmbroker.Broker wraps the same engine in a blocking API for plain scripts.
• Pluggable backends — swap registry, state store, and telemetry independently: DB, Redis etc.
• API keys configurable storage — environment variables, DB, AWS, Vault, your own store.
• Multi-user mode -- optionally separate secrets and LLMs list for each application user.
• Self-regulating pool — rate a reply's quality and llmbroker learns, per kind of task, which models are actually worth routing to; a consistently weak model quietly drops to the back of the queue instead of being nagged about by a human.

Documentation

llmbroker

Developers

Do not forget to run . ./activate.sh.

For work it need uv installed.

Use pre-commit hooks for code quality:

pre-commit install

Allure test report

• Allure report

Scripts

Install invoke preferably with uv tool:

uv tool install invoke

For a list of available scripts run:

invoke --list

For more information about a script run:

invoke <script> --help

The bundled freetier preset drifts as providers change their free tiers; refresh it with invoke catalog-refresh, which prints the maintenance runbook (presets/freetier-refresh-prompt.md).

Coverage report

• Codecov
• Coveralls

Created with cookiecutter using template