rolter

rolter

A high-performance, open-source LiteLLM-proxy alternative in Rust —
an OpenAI/Anthropic-compatible AI gateway and load balancer.

CI Release Documentation Coverage baseline: 64%

Latest release crates.io PyPI MSRV: Rust 1.82 License

rolter proxies commercial providers and load-balances self-hosted OpenAI-compatible fleets (e.g. 20–30 vLLM instances) with cache-aware routing, full RBAC, reload-free configuration, and cost/usage tracking.

Status: active development. The gateway, Postgres-backed control plane, reload-free configuration, cost controls, reliability primitives, and core provider surfaces are implemented; remaining work is tracked in ROADMAP.md, TODO.md, and GitHub issues.

Why rolter

  • Fast — a Rust data plane (Axum/Hyper/Tower on Tokio) with lock-free config reads and minimal-copy streaming.
  • Cache-aware load balancing — route prefix-heavy traffic to the vLLM replica most likely to have the KV cache warm.
  • Drop-in — speak the OpenAI and Anthropic APIs your clients already use.
  • Operable — virtual keys, budgets, rate limits, cost tracking, RBAC, and reload-free config changes from the UI.

Quick start — launch, configure, call

Go from zero to a working AI gateway in under a minute. The built-in fake-llm model answers locally, so the first request needs no provider key or config.

1. Launch

# single image: gateway + dashboard, no compose or config file
docker pull ghcr.io/rolter-ai/rolter:latest
docker run --rm -p 4000:4000 -p 4001:4001 ghcr.io/rolter-ai/rolter:latest

# native binary (installed from a release, cargo, uv, or pip)
rolter easy-up

Open the dashboard at http://localhost:4001. For Postgres, Redis, and ClickHouse, use the full-stack option instead:

docker compose -f docker/docker-compose.yml up -d

2. Configure

The dashboard is ready at http://localhost:4001. Add real providers and routes there when running in database mode, or use the bundled rolter.toml as your file-backed bootstrap config.

3. Call

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"fake-llm","messages":[{"role":"user","content":"hello"}]}'

Install methods, the rolter CLI reference, and production configuration are in the documentation.

Architecture

flowchart LR
  Client([OpenAI / Anthropic clients]) -->|/v1/*| GW["rolter-gateway<br/>(data plane)"]
  Admin([Dashboard]) --> CTL["rolter-control<br/>(control plane + UI host)"]
  GW -->|balanced + streamed| UP["Upstreams<br/>OpenAI · Anthropic · vLLM pool"]
  CTL -->|writes config| PG[("PostgreSQL")]
  CTL -->|publishes change events| RDS[("Redis")]
  RDS -->|hot-swap snapshot| GW
  GW -->|async batched logs| CH[("ClickHouse")]

Inspirations & Acknowledgments

rolter stands on the shoulders of great open-source projects and research. See our inspiration issues for detailed analysis of each project.

Gateway & Load Balancing

  • LiteLLM — proxy feature breadth, provider coverage (100+), config/DB model split, virtual keys, budget controls, spend tracking
  • Bifrost — high-performance Go gateway, weighted key selection (~10ns), multi-provider failover, plugin/middleware system, hierarchical budgets, semantic caching; published perf target (~11µs latency at 5k RPS)
  • TensorZero — Rust LLMOps gateway, sub-1ms p99 latency target at 10k+ QPS, observability patterns, OTLP traces + Prometheus, rate limiting with granular scopes
  • llm-d — cache-aware routing, prefix/KV-cache affinity, inference-phase scheduling, predicted-latency scheduling (40% TTFT/ITL reduction); highest-signal reference for rolter-balancer crate
  • LLMGateway — cost tracking, analytics dashboard UX, performance analytics, provider key management, self-host story (Docker + Postgres + Redis)
  • Archestra — dynamic model routing, MCP gateway, enterprise auth (OIDC, SAML, Okta, Entra), SSO + RBAC, tool-call safety guardrails

Infrastructure & Frameworks

  • vLLM — KV-cache-aware replica pooling and prefill/decode scheduling
  • Axum — high-performance Rust web framework
  • Tokio — async runtime foundation
  • shadcn/ui — component library for the dashboard UI

API Standards

Documentation

Repository layout

  • crates/rolter-core — config model, domain types, errors, telemetry
  • crates/rolter-balancer — load-balancing strategies (incl. approximate cache-aware)
  • crates/rolter-proxy — upstream forwarding, header injection, streaming
  • crates/rolter-store — repository traits + in-memory store (Postgres/Redis/ClickHouse next)
  • crates/rolter-auth — virtual keys, roles, access checks
  • crates/rolter-gateway — data-plane binary
  • crates/rolter-control — control-plane binary + static UI host
  • crates/rolter — unified rolter launcher (gateway / control / easy-up)
  • ui/ — Vite + React + shadcn/ui dashboard
  • docs/, user-docs/ — architecture/ADRs and the user documentation site
  • migrations/, clickhouse/ — database schemas

Development

cargo build --workspace
cargo nextest run --workspace   # tests via nextest (as CI does); + `cargo test --doc --workspace`
cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings

Commits and PR titles follow Conventional Commits. See AGENTS.md and docs/development/contributing.md.

License

Apache-2.0 — see LICENSE.

Project links

Data verified by PyPI on Aug 14, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for toiletsandpaper from gravatar.com toiletsandpaper

GitHub Statistics

Data verified by PyPI on Aug 14, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache-2.0

Requires

Python >=3.9

Tags

llm proxy gateway openai anthropic vllm load-balancer

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

rolter-0.0.11-py3-none-win_amd64.whl (9.0 MB view details)

Uploaded Python 3Windows x86-64

rolter-0.0.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.6 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

rolter-0.0.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (8.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

rolter-0.0.11-py3-none-macosx_11_0_arm64.whl (8.1 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

rolter-0.0.11-py3-none-macosx_10_12_x86_64.whl (8.4 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file rolter-0.0.11-py3-none-win_amd64.whl.

File metadata

  • Download URL: rolter-0.0.11-py3-none-win_amd64.whl
  • Upload date:
  • Size: 9.0 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for rolter-0.0.11-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 be34c124d4a09ea58db614eb327e9abcfce5d4df36ac233fd92b4671d12ee0af
MD5 b7cabb3e0f037479083a9e7b19e14137
BLAKE2b-256 62585c5c4bf0762c21207994cec47b29bfc5cccc9972771a9c687ae3f344a616

See more details on using hashes here.

Provenance

The following attestation bundles were made for rolter-0.0.11-py3-none-win_amd64.whl:

Publisher: release.yml on rolter-ai/rolter

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rolter-0.0.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for rolter-0.0.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4f9e577089f91605f2aaba9002875ba382a6286e31d20ba88faabfaf1301d148
MD5 e2ed04987bb236d7915df8b907ceee87
BLAKE2b-256 127e0cd31af281947fd71b77a98d66a5857258e531262ac7e397f19c66c9ec5e

See more details on using hashes here.

Provenance

The following attestation bundles were made for rolter-0.0.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on rolter-ai/rolter

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rolter-0.0.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for rolter-0.0.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b922780c3d500bd2e5fc5c73264a96554bb668c3337ce7170a88e8e7e013bb0b
MD5 2c76050f5ee64b8cb62b5339c2fc382f
BLAKE2b-256 e25c25cce451d06f78fdc323954d4ee0da925838a71e5cf0ca809cb5e667b2e5

See more details on using hashes here.

Provenance

The following attestation bundles were made for rolter-0.0.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on rolter-ai/rolter

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rolter-0.0.11-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for rolter-0.0.11-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6cec30f85897dc65203f23839ec7c6e679117a0e309364c9762e670c3059b143
MD5 cc160599f9b31d64d31e5c00d96e3d25
BLAKE2b-256 aab614fde37131b00ce23409d08a0e9e76cf0debc7950c996abac406e0fe74aa

See more details on using hashes here.

Provenance

The following attestation bundles were made for rolter-0.0.11-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on rolter-ai/rolter

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rolter-0.0.11-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for rolter-0.0.11-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 cb0fb68a0f7b038ef87a0dbc44000863f900553ec177a8ab806bd9a6da9a1dcb
MD5 ffa8901b815e3e1496a8987eb12141dd
BLAKE2b-256 98d4ee7fcb75f36be952280a9643ca1a7530b5f990a9bc47cd1e2c0668ad86b6

See more details on using hashes here.

Provenance

The following attestation bundles were made for rolter-0.0.11-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on rolter-ai/rolter

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page