llmbroker 0.0.4

No project description provided

llmbroker

Route LLM calls over a pool of free endpoints with automatic round-robin and 429/503 cooldown. No LangChain, no heavy deps — stdlib core, two lines to start.

import llmbroker

llms = llmbroker.Broker(registry=llmbroker.Registry("llms.toml"))
print(llms.ask("Summarize this receipt").text)

llms.toml is a plain list of [[llms]] entries (base_url, model, api_key_ref). Grab the freetier preset from this repo to start with a maintained list of free LLM endpoints.

Why llmbroker:

• Round-robin with automatic failover — 429/503 cools an endpoint and tries the next; the caller never sees a rate-limit error unless every endpoint fails.
• No heavy deps — stdlib-only core; optional backends (sqlite, redis, postgres) are submodules you import only when you need them.
• Dead-simple to start — a TOML file and env vars, one constructor line, done.
• Fully customisable — swap registry, secrets, and telemetry backends independently: any DB, any secrets manager (AWS, Vault, …), any storage.
• Cluster-ready — add shared_state=llmbroker.redis.SharedState(...) to sync cooldown state across instances; omit for single-process.
• Sync and async — llmbroker.Broker for scripts; llmbroker.AsyncBroker for FastAPI, agents, and async workers.

Documentation

llmbroker

Developers

Do not forget to run . ./activate.sh.

For work it need uv installed.

Use pre-commit hooks for code quality:

pre-commit install

Allure test report

• Allure report

Scripts

Install invoke preferably with uv tool:

uv tool install invoke

For a list of available scripts run:

invoke --list

For more information about a script run:

invoke <script> --help

Coverage report

• Codecov
• Coveralls

Created with cookiecutter using template