LiteLLM Routing Testbed

This package proves LiteLLM's context-window-aware routing locally on macOS without starting real vLLM instances. Two FastAPI dummy backends expose the OpenAI Chat Completions API, count every request with LiteLLM's own public token counter, and record one JSON object per received request.

What it proves

• Instance A accepts at most 65,000 input tokens.
• Instance B accepts at most 256,000 input tokens.
• LiteLLM pre-call checks remove deployments whose max_input_tokens is too small before choosing a backend.
• The native mode load-balances requests that fit both deployments.
• The deterministic mode uses LiteLLM deployment order to prefer A and uses B only after A is removed by the context check.
• A request with 256,001 input tokens is rejected by LiteLLM without reaching either backend.

Requirements

• Apple Silicon macOS
• Python 3.11 or newer
• Free local ports 4000, 4011, and 4012 by default

All services run on loopback and require no external API, database, Redis, container runtime, or model weights. The package forces LiteLLM to use its bundled model-cost map, so startup does not fetch the remote cost-map copy.

Install

cd litellm-routing-testbed
python3 -m venv .venv
.venv/bin/pip install -e .

For development and verification:

.venv/bin/pip install -e '.[dev]'

All direct dependencies are exactly pinned in pyproject.toml.

Run the native proof

The shortest complete run starts all services, executes the required target matrix plus 12 small-request probes, and always shuts down:

.venv/bin/routing-testbed all --mode native

Run lifecycle stages separately when inspecting live services:

.venv/bin/routing-testbed up --mode native
.venv/bin/routing-testbed status --mode native
.venv/bin/routing-testbed test --mode native
.venv/bin/routing-testbed down --mode native

Run deterministic small-first routing

.venv/bin/routing-testbed all --mode deterministic --small-probes 0

This mode adds order: 1 to A and order: 2 to B. The context check still runs first, so requests through 65k select A and larger valid requests select B.

Custom ports and target sizes

Every command accepts the same network and runtime options:

.venv/bin/routing-testbed all \
  --small-port 5011 \
  --large-port 5012 \
  --proxy-port 5000 \
  --targets 10000,50000,64000,70000,120000,200000,255000,256001

When commands are run separately, repeat the same options so they address the same state file and endpoints. Use --logs-dir and --run-dir to relocate runtime artifacts.

Endpoints

The two dummies expose:

• GET /health
• GET /v1/models
• POST /v1/chat/completions

LiteLLM listens on http://127.0.0.1:4000 by default. The public test model is router-model.

Evidence

Runtime evidence remains after down:

• logs/native-instance_A_65k.jsonl
• logs/native-instance_B_256k.jsonl
• logs/deterministic-instance_A_65k.jsonl
• logs/deterministic-instance_B_256k.jsonl
• logs/instance_A_65k.log
• logs/instance_B_256k.log
• logs/litellm.log
• logs/test-results-native.json
• logs/test-results-deterministic.json

Each backend JSONL row contains the UTC timestamp, instance, target port, client-reported target tokens, independently counted input tokens, max_tokens, model, and request ID. The assistant response content also contains handled_by and input_tokens, which the client uses for its pass/fail table. Mode-scoped filenames preserve native and deterministic evidence when the two runs are executed consecutively.

Cleanup

all calls down in a finally block. If a terminal is interrupted during a separate run, execute:

.venv/bin/routing-testbed down

The process manager checks stored command signatures before signaling PIDs, so a stale state file cannot cause it to terminate an unrelated reused PID.

See ROUTING.md for the verified LiteLLM fields and routing semantics.