rolter

A high-performance, open-source LiteLLM-proxy alternative in Rust —
an OpenAI/Anthropic-compatible AI gateway and load balancer.

rolter proxies commercial providers and load-balances self-hosted OpenAI-compatible fleets (e.g. 20–30 vLLM instances) with cache-aware routing, full RBAC, reload-free configuration, and cost/usage tracking.

Status: active development. The gateway, Postgres-backed control plane, reload-free configuration, cost controls, reliability primitives, and core provider surfaces are implemented; remaining work is tracked in ROADMAP.md, TODO.md, and GitHub issues.

Why rolter

• Fast — a Rust data plane (Axum/Hyper/Tower on Tokio) with lock-free config reads and minimal-copy streaming.
• Cache-aware load balancing — route prefix-heavy traffic to the vLLM replica most likely to have the KV cache warm.
• Drop-in — speak the OpenAI and Anthropic APIs your clients already use.
• Operable — virtual keys, budgets, rate limits, cost tracking, RBAC, and reload-free config changes from the UI.

Quick start — launch, configure, call

Go from zero to a working AI gateway in under a minute. The built-in fake-llm model answers locally, so the first request needs no provider key or config.

1. Launch

# single image: gateway + dashboard, no compose or config file
docker pull ghcr.io/rolter-ai/rolter:latest
docker run --rm -p 4000:4000 -p 4001:4001 ghcr.io/rolter-ai/rolter:latest

# native binary (installed from a release, cargo, uv, or pip)
rolter easy-up

Open the dashboard at http://localhost:4001. For Postgres, Redis, and ClickHouse, use the full-stack option instead:

docker compose -f docker/docker-compose.yml up -d

2. Configure

The dashboard is ready at http://localhost:4001. Add real providers and routes there when running in database mode, or use the bundled rolter.toml as your file-backed bootstrap config.

3. Call

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"fake-llm","messages":[{"role":"user","content":"hello"}]}'

Install methods, the rolter CLI reference, and production configuration are in the documentation.

Architecture

flowchart LR
  Client([OpenAI / Anthropic clients]) -->|/v1/*| GW["rolter-gateway<br/>(data plane)"]
  Admin([Dashboard]) --> CTL["rolter-control<br/>(control plane + UI host)"]
  GW -->|balanced + streamed| UP["Upstreams<br/>OpenAI · Anthropic · vLLM pool"]
  CTL -->|writes config| PG[("PostgreSQL")]
  CTL -->|publishes change events| RDS[("Redis")]
  RDS -->|hot-swap snapshot| GW
  GW -->|async batched logs| CH[("ClickHouse")]

Inspirations & Acknowledgments

rolter stands on the shoulders of great open-source projects and research. See our inspiration issues for detailed analysis of each project.

Gateway & Load Balancing

• LiteLLM — proxy feature breadth, provider coverage (100+), config/DB model split, virtual keys, budget controls, spend tracking
• Bifrost — high-performance Go gateway, weighted key selection (~10ns), multi-provider failover, plugin/middleware system, hierarchical budgets, semantic caching; published perf target (~11µs latency at 5k RPS)
• TensorZero — Rust LLMOps gateway, sub-1ms p99 latency target at 10k+ QPS, observability patterns, OTLP traces + Prometheus, rate limiting with granular scopes
• llm-d — cache-aware routing, prefix/KV-cache affinity, inference-phase scheduling, predicted-latency scheduling (40% TTFT/ITL reduction); highest-signal reference for rolter-balancer crate
• LLMGateway — cost tracking, analytics dashboard UX, performance analytics, provider key management, self-host story (Docker + Postgres + Redis)
• Archestra — dynamic model routing, MCP gateway, enterprise auth (OIDC, SAML, Okta, Entra), SSO + RBAC, tool-call safety guardrails

Infrastructure & Frameworks

• vLLM — KV-cache-aware replica pooling and prefill/decode scheduling
• Axum — high-performance Rust web framework
• Tokio — async runtime foundation
• shadcn/ui — component library for the dashboard UI

API Standards

• OpenAI and Anthropic — API compatibility targets and standards

Documentation

• Quickstart and Installation — install methods and the unified rolter CLI (gateway / control / easy-up)
• Configuration, Deployment, and Observability guides
• Air-gapped install & operation — running fully offline behind an internal mirror
• Architecture overview — the full design and ADRs

Repository layout

• crates/rolter-core — config model, domain types, errors, telemetry
• crates/rolter-balancer — load-balancing strategies (incl. approximate cache-aware)
• crates/rolter-proxy — upstream forwarding, header injection, streaming
• crates/rolter-store — repository traits + in-memory store (Postgres/Redis/ClickHouse next)
• crates/rolter-auth — virtual keys, roles, access checks
• crates/rolter-gateway — data-plane binary
• crates/rolter-control — control-plane binary + static UI host
• crates/rolter — unified rolter launcher (gateway / control / easy-up)
• ui/ — Vite + React + shadcn/ui dashboard
• docs/, user-docs/ — architecture/ADRs and the user documentation site
• migrations/, clickhouse/ — database schemas

Development

cargo build --workspace
cargo nextest run --workspace   # tests via nextest (as CI does); + `cargo test --doc --workspace`
cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings

Commits and PR titles follow Conventional Commits. See AGENTS.md and docs/development/contributing.md.

License

Apache-2.0 — see LICENSE.