llm-cascade

Free-first LLM routing for Python: local Ollama ($0) -> OpenRouter free models ($0) -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.

Why

Most teams route every prompt to the same frontier model regardless of how hard the task actually is, and the API bill grows with usage instead of with value delivered. llm-cascade flips that: it tries the cheapest tier that can plausibly do the job first, and only escalates when that tier is unreachable or genuinely too weak for the task. In production, that means classification, extraction, and summarization run for $0 on a local model or a free cloud model, while code review, planning, and long-form writing still get a frontier model -- and if the paid API is ever down or out of credits, your scheduled jobs degrade to a free tier instead of crashing.

Install

pip install llm-cascade

The base package has zero required third-party dependencies. To use the paid Anthropic tier, install the extra:

pip install "llm-cascade[anthropic]"

Without it, the local (Ollama) and free-cloud (OpenRouter) tiers still work end to end -- you'll only see an error if a task actually routes to the paid tier and the SDK isn't installed.

Quickstart

from llm_cascade import route

result = route("summarize", "Paste a long document here...")
print(result.text)        # the completion
print(result.tier)        # which tier served it: trivial/local/cheap/light/heavy
print(result.model)       # e.g. "local:ollama", "openrouter:qwen/...", "claude-sonnet-5"
print(result.cost_usd)    # 0.0 for local/free-cloud tiers

Or from the command line:

llm-cascade "Summarize this text: ..." --task summarize
llm-cascade --routes   # show the tier ladder and task -> tier table
llm-cascade --stats    # show cumulative cost and free-vs-paid call counts

How routing works

Each call resolves a task name (or a tier name directly) to a tier, then runs that tier's provider chain in order until one succeeds:

If the paid API call fails for any reason -- bad or revoked key, no credits, a provider outage, a network blip -- llm-cascade degrades to the free tiers instead of raising, so scheduled or automated callers don't break outright. It only raises once every tier has been tried and failed.

The default task -> tier table:

classify, extract, summarize, translate, chat  -> local
rewrite                                        -> light
code, review, plan, creative                   -> heavy

Anything not in that table falls back to default_task_tier (light by default). See config/routes.example.toml for the full, commented schema.

Configuration

Configuration is resolved with this precedence (lowest to highest):

1. Built-in defaults.
2. An optional TOML file: ~/.llm_cascade/routes.toml, or the path given in LLM_CASCADE_CONFIG.
3. Environment variables, for the handful of settings that make sense as env vars.

Copy config/routes.example.toml to ~/.llm_cascade/routes.toml to add your own task names, retune which tier handles a task, or override model ids and pricing.

Environment variables:

See .env.example for a copy-pasteable starting point.

Savings ledger

Every call -- free or paid -- appends one line to a JSONL ledger (~/.llm_cascade/log.jsonl by default). stats() / llm-cascade --stats summarize it:

$ llm-cascade --stats
{
  "paid_calls": 3,
  "free_calls": 41,
  "total_cost_usd": 0.0842,
  "by_model": {
    "local:ollama": 30,
    "openrouter:qwen/qwen3-next-80b-a3b-instruct:free": 11,
    "claude-sonnet-5": 3
  }
}

Comparison

Unlike heavier multi-provider routers, llm-cascade ships with zero required dependencies and a single opinionated cascade -- it's a routing policy you can read in one file, not a platform.

Contributing

Issues and pull requests are welcome at github.com/luandv92/llm-cascade. Before opening a PR:

pip install -e ".[dev,anthropic]"
ruff check .
pytest tests/ -v

Tests never touch real networks -- Ollama, OpenRouter, and the Anthropic SDK are all mocked (see tests/conftest.py).

License

MIT -- see LICENSE.