theagentrouter

Call AI models from your agent with LiteLLM's API.
Every call goes through Agent Router, an open-source AI gateway, which handles fallbacks, retries and request limits.

Alpha: names and behaviour may change before 1.0.

Quick start

pip install theagentrouter
export ANTHROPIC_API_KEY=...

import theagentrouter

resp = theagentrouter.completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Summarise yesterday's refunds"}],
)
print(resp.choices[0].message.content)

The first call starts a local gateway on your computer, and the gateway calls the provider with your key. On Windows, Intel Macs and older Linux there is no local gateway; use a remote one instead (see Install).

Why use it

• Already using LiteLLM? Change one import. Functions, arguments, responses and errors stay the same.
• Fallbacks and retries run in a gateway, not in your process. Define a route once, and every program on the computer uses it.
• Google ADK and the OpenAI Agents SDK work without LiteLLM installed.
• Point it at your company's gateway with two environment variables. Provider keys then stay on that gateway, not on laptops.

Examples

All twenty-one, with the keys each one needs: examples/.

Coming from LiteLLM

from theagentrouter.compat import litellm   # was: import litellm

Change this one import; the rest of your code stays the same. Some LiteLLM functions are not in the SDK, and some arguments are ignored with a warning. The full list: docs/coming-from-litellm.md.

Fallbacks and retries

A route is a model name you choose, with the models behind it in order. The local gateway calls the first model, retries it if it fails, then moves on to the next one:

import theagentrouter as ar

ar.route("chat", primary="openai/gpt-4o", fallbacks=["anthropic/claude-sonnet-4-5"], retry=ar.Retry(attempts=2))
ar.completion(model="chat", messages=[{"role": "user", "content": "hi"}])

Here a call to chat goes to openai/gpt-4o. If it cannot be reached or answers with 429, 500, 502, 503 or 504, the gateway retries it up to 2 more times, then tries anthropic/claude-sonnet-4-5 the same way. Other errors, such as 401, are not retried.

• A route can also limit requests: limit=ar.Limit(60, per="minute") lets 60 requests a minute through and answers the rest with a RateLimitError.
• Routes are saved in ~/.agentrouter/routes.json, so every program on this computer can use them.
• Routes work with the local gateway only. With AGENTROUTER_BASE_URL set, set up fallbacks on that gateway instead.
• Router from theagentrouter.compat.litellm, with LiteLLM's fallbacks and num_retries, is turned into routes.

Details: Routes.

Agent frameworks

install_as_litellm() is in theagentrouter.compat. It makes import litellm load this SDK, so neither framework needs LiteLLM installed. Details and examples: docs/frameworks.md.

Install

pip install theagentrouter

This installs the SDK and the local gateway. The local gateway is Agent Router's aigw program, about 80 MB, and it runs on:

• macOS 13 or newer on Apple chips
• Linux with glibc 2.36 or newer, on x86-64 or Arm

On any other computer (Windows, Intel Macs, macOS 12 or older, Alpine Linux, Linux with glibc older than 2.36) there is no local gateway, so the same command installs the SDK alone. Calls then need a remote gateway (below).

If your code will use a remote gateway, for example on a server, in a container, in a serverless function or in CI, install the SDK alone and skip the 80 MB:

pip install theagentrouter-core

Both packages are imported the same way: import theagentrouter.

Using a remote gateway. To send calls to an Agent Router gateway that your team or company runs, set its address and key:

export AGENTROUTER_BASE_URL=https://gateway.acme.example/v1
export AGENTROUTER_API_KEY=...

With these set, calls go to that gateway, the local gateway never starts, and the provider keys stay on the remote gateway. This works on every computer.

The local gateway keeps running after your program ends, and every program on this computer that uses the SDK shares it. Its files are in ~/.agentrouter/local-gateway/ (or AGENTROUTER_HOME/local-gateway/): config.yaml, state.json and aigw.log. To stop it (the next call starts it again):

python -c "from theagentrouter import local_gateway; local_gateway.stop()"

Installing from source before the first release, checking the install, upgrading and removing: docs/install.md.

Documentation

For coding agents

• llms.txt and llms-full.txt: the docs and examples in the llms.txt format, for coding agents and AI search.
• skills/: agent skills (SKILL.md). migrate-from-litellm moves a project off LiteLLM; theagentrouter writes new code with the SDK. Copy a folder into your agent's skills folder (for Claude Code, .claude/skills/), then ask it to "migrate this project from LiteLLM to theagentrouter".

More

• DEVELOPMENT.md: what is where in this repository, setting up, tests, the framework tests, real-provider checks, releases.
• CONTRIBUTING.md, SECURITY.md, CHANGELOG.md.

Design

1. Only what agent frameworks use. The SDK has a LiteLLM function or setting only if an agent framework calls it. So there is no image generation, speech, video, files or batches.
2. Agent Router handles the calls. The gateway does fallbacks, retries, request limits, provider keys and logging. The SDK only does what a gateway can't, like building LiteLLM's response objects or running your callbacks.
3. Same API as LiteLLM. Same function names, arguments, responses and errors.

License

Apache License 2.0.