RouteWise

Latency–Cost Optimization for Multi-Provider LLM Routing
_{Learn from recent outcomes. Route each request. Hedge selectively.}

Developed and maintained by the Harvard MadSys Lab at Harvard SEAS.

English API · 中文 API · Reproducibility

RouteWise is a dependency-free Python library for cost-aware, latency-optimized routing across multiple LLM API providers. Applications supply provider prices, dispatch the returned attempt, and report outcomes so RouteWise can learn from them.

The 0.2.0 distribution is an API-provider-only preview. This repository also contains the simulator and experiment harnesses used by the paper; those research packages are not included in the wheel.

Package name: The PyPI project routewise is an unaffiliated, incompatible project. Install the llm-routewise distribution and import the llm_routewise package for HarvardMadSys RouteWise.

Installation

RouteWise requires Python 3.10 or later. The published wheel has no runtime dependencies.

python -m pip install llm-routewise==0.2.0

For repository development and paper-artifact workflows:

git clone https://github.com/HarvardMadSys/RouteWise.git
cd RouteWise
uv sync

Quickstart

import llm_routewise as rw

router = rw.Router(
    [
        rw.Provider("fast", price_in=3.0, price_out=15.0),
        rw.Provider("cheap", price_in=0.15, price_out=0.60),
    ],
    alpha=0.25,  # Cost budget: 0 = cheapest; 1 = full range for latency optimization.
)

decision = router.route(input_tokens=800)
response = call_your_provider(decision.provider)
decision.completed(
    ttft_ms=response.ttft_ms,
    output_tokens=response.output_tokens,
)

If your application already predicts completion length, pass that point estimate with the request. Omit it to use RouteWise's internal online estimate.

predicted_tokens = predict_output_tokens(prompt)
decision = router.route(
    input_tokens=800,
    estimated_output_tokens=predicted_tokens,
)

The estimate affects route and hedge cost calculations only; it is not actual usage. On completion, report the adopted attempt's actual output_tokens (or an explicit cost_usd) for billing. Positive actual output tokens also update RouteWise's internal estimator.

Router computes decisions but performs no network I/O and does not read API keys. Your application owns provider clients, credentials, and dispatch. Read the English API reference or 中文 API 参考 for the full contract.

Repository Development

Run the fast test suite:

uv run pytest -q -m "not slow"

List the paper-facing simulator sections:

uv run python -m routewise_cli.main simulator list

The reproducibility guide covers datasets, live-evaluation credentials, experiment commands, and regression checks.

Documentation

Library Users

• Python API
• Python API, Chinese

Research Artifacts

• Simulator architecture and algorithms
• Experiment reproducibility

Maintainers and Advanced Integrators

• Core mathematical API
• Release procedure
• Published-package changes

License

MIT. See the license.