llm-cascade 0.1.0

Free-first LLM routing for Python: local Ollama -> OpenRouter free models -> paid Anthropic API, with automatic fallback and a savings ledger.

llm-cascade

Free-first LLM routing for Python: local Ollama ($0) -> OpenRouter free models ($0) -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.

Why

Most teams route every prompt to the same frontier model regardless of how hard the task actually is, and the API bill grows with usage instead of with value delivered. llm-cascade flips that: it tries the cheapest tier that can plausibly do the job first, and only escalates when that tier is unreachable or genuinely too weak for the task. In production, that means classification, extraction, and summarization run for $0 on a local model or a free cloud model, while code review, planning, and long-form writing still get a frontier model -- and if the paid API is ever down or out of credits, your scheduled jobs degrade to a free tier instead of crashing.

Install

pip install llm-cascade

The base package has zero required third-party dependencies. To use the paid Anthropic tier, install the extra:

pip install "llm-cascade[anthropic]"

Without it, the local (Ollama) and free-cloud (OpenRouter) tiers still work end to end -- you'll only see an error if a task actually routes to the paid tier and the SDK isn't installed.

Quickstart

from llm_cascade import route

result = route("summarize", "Paste a long document here...")
print(result.text)        # the completion
print(result.tier)        # which tier served it: trivial/local/cheap/light/heavy
print(result.model)       # e.g. "local:ollama", "openrouter:qwen/...", "claude-sonnet-5"
print(result.cost_usd)    # 0.0 for local/free-cloud tiers

Or from the command line:

llm-cascade "Summarize this text: ..." --task summarize
llm-cascade --routes   # show the tier ladder and task -> tier table
llm-cascade --stats    # show cumulative cost and free-vs-paid call counts

How routing works

Each call resolves a task name (or a tier name directly) to a tier, then runs that tier's provider chain in order until one succeeds:

Tier	Chain	Use for
trivial	local Ollama only ($0)	high-volume micro-tasks, never worth the network
local	local Ollama ($0) -> OpenRouter free ($0) -> paid	mechanical NLP: classify, extract, summarize, translate
cheap	OpenRouter free ($0) -> paid	manual opt-in tier, invoke directly with task="cheap"
light	paid (Sonnet); free-cloud only if LLM_CASCADE_LIGHT_FREE=1	customer-facing copy that needs reliable polish
heavy	paid (Opus), adaptive thinking, high effort	real reasoning: code, review, planning, long-form

If the paid API call fails for any reason -- bad or revoked key, no credits, a provider outage, a network blip -- llm-cascade degrades to the free tiers instead of raising, so scheduled or automated callers don't break outright. It only raises once every tier has been tried and failed.

The default task -> tier table:

classify, extract, summarize, translate, chat  -> local
rewrite                                        -> light
code, review, plan, creative                   -> heavy

Anything not in that table falls back to default_task_tier (light by default). See config/routes.example.toml for the full, commented schema.

Configuration

Configuration is resolved with this precedence (lowest to highest):

1. Built-in defaults.
2. An optional TOML file: ~/.llm_cascade/routes.toml, or the path given in LLM_CASCADE_CONFIG.
3. Environment variables, for the handful of settings that make sense as env vars.

Copy config/routes.example.toml to ~/.llm_cascade/routes.toml to add your own task names, retune which tier handles a task, or override model ids and pricing.

Environment variables:

Variable	Purpose
ANTHROPIC_API_KEY	Enables the paid tier (light/heavy, and the final safety net).
OPENROUTER_API_KEY	Enables the free-cloud tier (OpenRouter :free models).
LLM_CASCADE_CONFIG	Path to a TOML config file, instead of ~/.llm_cascade/routes.toml.
LLM_CASCADE_LOG_PATH	Path to the savings ledger, instead of ~/.llm_cascade/log.jsonl.
LLM_CASCADE_LIGHT_FREE	Set to 1 to try free-cloud first on the light tier too.
LLM_CASCADE_NO_FREE_CLOUD	Set to any value to disable the free-cloud tier globally.
LLM_CASCADE_HTTP_REFERER / LLM_CASCADE_APP_TITLE	Sent as HTTP-Referer / X-Title on OpenRouter requests (optional; omitted if unset).
LLM_CASCADE_OPENROUTER_MODEL / LLM_CASCADE_OPENROUTER_MODELS	Override the free model (singular) or ordered list (comma-separated) tried on OpenRouter.
LLM_CASCADE_OLLAMA_BASE_URL	Ollama server URL (default http://localhost:11434).

See .env.example for a copy-pasteable starting point.

Savings ledger

Every call -- free or paid -- appends one line to a JSONL ledger (~/.llm_cascade/log.jsonl by default). stats() / llm-cascade --stats summarize it:

$ llm-cascade --stats
{
  "paid_calls": 3,
  "free_calls": 41,
  "total_cost_usd": 0.0842,
  "by_model": {
    "local:ollama": 30,
    "openrouter:qwen/qwen3-next-80b-a3b-instruct:free": 11,
    "claude-sonnet-5": 3
  }
}

Comparison

Unlike heavier multi-provider routers, llm-cascade ships with zero required dependencies and a single opinionated cascade -- it's a routing policy you can read in one file, not a platform.

Contributing

Issues and pull requests are welcome at github.com/luandv92/llm-cascade. Before opening a PR:

pip install -e ".[dev,anthropic]"
ruff check .
pytest tests/ -v

Tests never touch real networks -- Ollama, OpenRouter, and the Anthropic SDK are all mocked (see tests/conftest.py).

License

MIT -- see LICENSE.