Quick start

cp .env.oss.example .env      # add whichever provider keys you have
docker compose up             # Postgres + Redis + gateway + admin console

Adapters activate based on which provider keys you set. GET /v1/models lists only what your box can serve; a single-provider deployment works without any extra configuration.

Don't want to run infrastructure? The managed deployment of this same codebase runs at mlpal.ai — create a key there and point the same SDK/CLI at https://models.mlpal.ai instead of localhost.

Calling it

Anthropic wire, any provider's model:

curl http://localhost:8000/v1/messages \
  -H "Authorization: Bearer $MLPAL_KEY" -H "content-type: application/json" \
  -d '{"model":"mlpal","max_tokens":512,"messages":[{"role":"user","content":"hi"}]}'

Existing OpenAI SDK code — point it at http://localhost:8000/v1. Python SDK — pip install mlpal-assistants:

from mlpal_assistants import MLPal

client = MLPal(base_url="http://localhost:8000")   # key from MLPAL_API_KEY
msg = client.messages.create(
    model="mlpal", max_tokens=512,
    messages=[{"role": "user", "content": "Hello"}],
)
print(msg.text, msg.compute_units)

What you get

• One wire format for every provider. POST /v1/messages speaks the Anthropic Messages format for all served models; the gateway translates to each provider's native API and relays the SSE stream without re-chunking. OpenAI-compatible endpoints (/v1/chat/completions, embeddings, images, audio) serve existing OpenAI SDK code unchanged.
• Router tags. "model": "mlpal" resolves to the best model your deployment serves, walking a curated candidate list that spans providers. Model retired? Provider key missing? Resolution falls through to the next candidate. Client code never changes.
• Cost on every response. Requests are metered in compute units (1 CU = $10 of provider list price, no markup) and returned in an X-MLPal-Compute-Units header. The meter reproduces provider list pricing exactly, including prompt-cache discounts.
• Per-key control. Model-policy globs (allow: ["claude-*"]), multi-window spend budgets, permissions — all enforced at admission, before the provider call. A running stream is never cut.
• Observability. Per-key cache hit rate, latency p50/p95, time-to-first-token, request traces, and optional payload capture (zlib-compressed, runtime toggle — your box, your data).
• Provider semantics preserved. Prompt caching (cache_control), tools, structured output, and MCP config pass through untouched.

Measured

Gateway overhead isolated against a zero-latency fake upstream, so provider variance can't hide anything — MLPal with its full admission pipeline (auth, rate limit, billing, model policy, budgets, metering, capture) against LiteLLM in both its bare mode and a production configuration with database-backed virtual keys, budgets, and spend tracking (N=100 per system; methodology in the technical report, raw data and harness in paper/bench/):

+8.5 ms with everything on — less than a bare proxy checks one static key for, and with the tightest tail (p95 33 ms vs 58/41 ms). Admission-time governance is computationally free. Provider semantics survive the hop too: a 22k-token cached prefix passes through byte-faithfully and metered 0.002756 CU on write, 0.000224 CU on read (12.3×) — matching Anthropic's list price to five decimals. The managed deployment of this same codebase, measured the same night from the same client, served claude-haiku-4.5 at 642 ms median TTFT vs OpenRouter's 898 ms (report §5.3–5.4).

Why a curated catalog

Production models retire on roughly a 12-month cycle now — from the providers' own deprecation ledgers:

Serving a model well — valid parameter ranges, per-model cache minimums, reasoning budgets, provider quirks — is per-model engineering that does not scale to a 1,600-entry catalog. This gateway serves a curated set (~75 models) kept current by a data feed, and router tags absorb every retirement above server-side. You can always pin any explicit model tag or register your own adapter.

The full argument with benchmarks and sources: Curation Over Breadth · feature matrix vs. OpenRouter / LiteLLM / Portkey: docs/POSITIONING.md.

Router tags vs. catalog

Two ways to use the curated set — they differ in who picks the model:

Both are driven by the same feed (catalog/*.json) and update as data, not code.

API surface

Details: docs/API_SURFACE.md.

Use it with a coding agent

Yodex is a coding CLI built on this gateway — it speaks the Anthropic wire and uses GET /v1/catalog to route sub-tasks to cheaper models (~10× lower sub-agent cost in its benchmarks):

npm install -g @mlpal/yodex
export YODEX_GATEWAY_URL=http://localhost:8000
export YODEX_API_KEY=mlpal_sk_...    # minted in the console
yodex "fix the failing test"

Repository layout

src/                 # FastAPI gateway: adapters, services, api, seams
console/             # admin UI (React + Vite): keys, traces, catalog, usage
docker-compose.yaml  # one-command local deployment
alembic/             # database migrations
paper/               # technical report + benchmark harness + raw results
enterprise/          # commercial add-ons (separate license, NOT Apache)
docs/                # API surface, positioning, figures

Auth and billing sit behind composition-root seams (api/mounting.py); the defaults (MLPAL_AUTH_BACKEND=local, MLPAL_BILLING_BACKEND=local) run fully standalone with no external dependencies. src/ never imports from enterprise/.

License and contact

Apache-2.0, except the enterprise/ directory (commercial — see enterprise/LICENSE). Contributions welcome: CONTRIBUTING.md · security and everything else: contact@mlpal.ai