llmbroker 0.0.10

No project description provided

llmbroker

Route LLM calls over a pool of free LLMs with automatic round-robin and cooldown.

No LangChain, no heavy deps.

import llmbroker

llms = llmbroker.Broker("llms.toml")
print(llms.ask("Summarize this receipt").text)

llms.toml is a plain list of [[llms]] entries (base_url, model, api_key_ref). Grab the freetier preset from this repo to start with a maintained list of free LLM endpoints.

That one-liner is the whole API for a script — but the same Broker scales up too. Run it inside a server or across a cluster with cooldown state shared between instances, and back it with common databases (connectors for SQLite, Redis, and Postgres covering registry, telemetry, secrets, and state). The calling code stays the same — you just pass a backend.

Why llmbroker:

• Round-robin with automatic failover — 429/503 cools an endpoint and tries the next; the caller never sees a rate-limit error unless every endpoint fails.
• No heavy deps — stdlib-only core; optional backends (sqlite, redis, postgres) are submodules you import only when you need them.
• Dead-simple to start — a TOML file and env vars, one constructor line, done.
• Fully customisable — swap registry, secrets, and telemetry backends independently: any DB, any secrets manager (AWS, Vault, …), any storage.
• Cluster-ready — add state_store=llmbroker.redis.StateStore(...) to sync cooldown state across instances; omit for single-process.
• Sync and async — llmbroker.Broker for scripts; llmbroker.AsyncBroker for FastAPI, agents, and async workers.

Documentation

llmbroker

Developers

Do not forget to run . ./activate.sh.

For work it need uv installed.

Use pre-commit hooks for code quality:

pre-commit install

Allure test report

• Allure report

Scripts

Install invoke preferably with uv tool:

uv tool install invoke

For a list of available scripts run:

invoke --list

For more information about a script run:

invoke <script> --help

Coverage report

• Codecov
• Coveralls

Created with cookiecutter using template