Skip to main content

warpllm

A warp-speed, robust AI gateway written for rust, node, and python applications - built for planet scale by the community.

Discord Reddit

CI crates.io PyPI npm License

Quickstart

pip install warpllm              # python
npm install @warpllm/warpllm     # node
cargo add warpllm                # rust
export OPENAI_API_KEY=sk-...

Python

from warpllm import WarpLLM

client = WarpLLM()

completion = client.chat_completions({
    "model": "openai/gpt-5-nano",
    "messages": [{"role": "user", "content": "Hello!"}],
})

print(completion["choices"][0]["message"]["content"])

Node

import { WarpLLM } from '@warpllm/warpllm'

const client = new WarpLLM()

const completion = await client.chatCompletions({
  model: 'openai/gpt-5-nano',
  messages: [{ role: 'user', content: 'Hello!' }],
})

console.log(completion.choices[0].message.content)

Rustchat_completions is async and warpllm ships no runtime, so bring your own: cargo add tokio --features macros,rt-multi-thread.

use warpllm::{ChatCompletionRequestMessage, Client, ClientConfig, CreateChatCompletionRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new(ClientConfig::default())?;

    let completion = client
        .chat_completions(CreateChatCompletionRequest {
            model: "openai/gpt-5-nano".to_string(),
            messages: vec![ChatCompletionRequestMessage {
                role: "user".to_string(),
                content: "Hello!".to_string(),
                ..Default::default()
            }],
            ..Default::default()
        })
        .await?;

    let content = completion.choices[0].message.content.as_deref();
    println!("{}", content.unwrap_or_default());
    Ok(())
}

Switching providers is a string change

Keys are read from the environment when the client is built, so export the one the provider needs and change the model string. Nothing else moves.

Model string Key it needs
openai/gpt-5-nano OPENAI_API_KEY
deepseek/deepseek-v4-flash DEEPSEEK_API_KEY
kimi/kimi-k3 MOONSHOT_API_KEY
opencode/glm-5.2 OPENCODE_API_KEY
openrouter/anthropic/claude-sonnet-4 OPENROUTER_API_KEY

The provider/ prefix is required. warpllm matches the whole string against its roster, so a bare gpt-5-nano — or any name it doesn't know — is an error rather than a guess at an upstream default.

Narrowing a client to the providers it serves

By default a client serves the whole roster and reads every provider's variable. Declare the ones you mean and it reads no others, routes to no others, and takes a key directly for the callers who keep theirs somewhere the environment can't reach:

WarpLLM(providers={"openai": {}, "deepseek": {"api_key": "sk-..."}})
new WarpLLM({ providers: { openai: {}, deepseek: { apiKey: 'sk-...' } } })
ClientConfig {
    providers: Some(BTreeMap::from([
        ("openai".into(), ProviderConfig::default()),
        ("deepseek".into(), ProviderConfig { api_key: Some(key) }),
    ])),
    ..Default::default()
}

An empty entry means "serve this one, key from the environment". A request for a model under a provider you didn't declare is refused before any upstream call, and a provider name the roster doesn't hold fails when the client is built.

Runnable versions of all three, with comments, are in examples/.

Your own models

Anything that speaks the OpenAI API — vLLM, TGI, Ollama, llama.cpp — is a routing target. Describe it in a file and hand warpllm the path:

# ./warpllm.yaml
providers:
  local:
    base_url: "http://localhost:8000/v1"
    auth: none                       # the box is on a private network
    models:
      local/llama-3.3-70b:
        supported_apis:
          - {api: openai_compat_chat_completions}
          - {api: openai_compat_chat_completions_stream}
client = WarpLLM(specs_path="./warpllm.yaml")
client.chat_completions({"model": "local/llama-3.3-70b", "messages": [...]})
const client = new WarpLLM({ specsPath: './warpllm.yaml' })
let client = Client::new(ClientConfig {
    specs_path: Some("./warpllm.yaml".into()),
    ..Default::default()
})?;
warpllm-server --specs ./warpllm.yaml   # or WARPLLM_SPECS=./warpllm.yaml

Your file is merged over the built-in roster, so adding local/ leaves openai/ exactly where it was — the same client routes both. Reusing a built-in provider's name replaces that provider whole, and warpllm warns rather than shadowing it quietly. The warning goes through tracing, which warpllm-server surfaces and a Rust client does once it installs a subscriber; the Python and Node bindings install none yet, so there it goes nowhere. Same for the older warning about an environment with no provider keys in it.

auth: none is the line that matters for a private box: warpllm then sends no Authorization header at all. Omitting it means something else — that the roster records no way to authenticate this provider — so a forgotten env_api_key on a paid provider fails locally instead of leaving without a credential.

The file is read when the client is built, so a roster that can't be used is an error there, naming the path — not a request failing hours later. There is no wildcard: every model gets an entry, because supported_apis and capabilities are per model and a pattern would have to claim both on behalf of models nobody listed.

The schema is documented in full at the top of specs.yaml, and examples/warpllm.yaml is a worked one covering vLLM, Ollama, and a cluster that does want a key.

Mission

This project is to lay out the most resilient open source productionization layer for AI-deployments. Designed for you if you want:

  1. To work with multiple AI providers or your own models.
  2. To keep your AI services up and running with 0 downtime.
  3. Speed (minimal overhead latency).
  4. A granular view of your metrics (uptime, P95 latency, costs, etc).
  5. Control over:
    1. Where your data goes.
    2. Your AI budget across providers.

Status

[!IMPORTANT] The published packages are 0.5.0, which lets a client bring its own roster file — so a self-hosted OpenAI-compatible server is a routable target without forking the crate — and adds weighted load balancing (Rust only) and Mistral. It is source-breaking for Rust only: ClientConfig gained a field, so an exhaustive struct literal no longer compiles — add specs_path: None, or switch to ..Default::default(). Python and TypeScript are purely additive. See the changelog before upgrading from 0.4.x.

The OpenAI-compatible HTTP gateway has landed on main but is not released yet.

Released (0.5.0) On main
OpenAI chat completions, non-streaming Yes Yes
provider/model routing strings Provider registry Provider registry
DeepSeek, OpenRouter Yes Yes
Kimi Yes Yes
Mistral Yes Yes
OpenCode Zen Yes Yes
Declaring the providers a client serves Yes Yes
Self-hosted models via your own roster file Yes Yes
OpenAI-compatible HTTP gateway Unreleased
Streaming Yes Yes
Weighted load balancing Rust only Rust only
Failover, caching, metrics

Unlisted models are rejected rather than guessed at, so routing a name warpllm doesn't know is an error, not a surprise upstream bill.

Layers

  1. An SDK - provide a request and we translate it to work with different providers and models out of box.
  2. [Unreleased] A proxy - run a self-hosted proxy that speaks the OpenAI API:
    1. [Coming Soon] Failover - define multiple models to handle outages / errors
    2. [Coming Soon] Load Balancing - define a % of requests to be handled per model
    3. [Coming Soon] Prompt Response Caching - define a TTL and avoid paying twice for the same prompt

Key focus points

  1. Native SDK support - Written once in rust, compiled for maximum performance, available for rust/typescript/python.
  2. Self hostable - Avoid vendor lock-in (e.g. from cloud provider or model provider), or data leaving your infra.
  3. Warp-speed execution - What we named ourselves after. Machine level code, faster than a typescript or python native library.
  4. Compact file size - Pre-compiled into binary format, not verbose text files.

Roadmap

The roadmap lives in GitHub issues — one issue per item, so direction is discussed where the work happens. Add a comment if you see something missing, or if something there matters enough to you that it should move up.

Contributing

We're excited to have you join us. See the contribution guide for how to get started.

A big thank you to the contributors below who have helped build this AI gateway to this point!

License

The warpllm core is open source under the Apache License 2.0.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

warpllm-0.5.0.tar.gz (435.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

warpllm-0.5.0-cp310-abi3-win_arm64.whl (2.4 MB view details)

Uploaded CPython 3.10+Windows ARM64

warpllm-0.5.0-cp310-abi3-win_amd64.whl (2.5 MB view details)

Uploaded CPython 3.10+Windows x86-64

warpllm-0.5.0-cp310-abi3-musllinux_1_2_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

warpllm-0.5.0-cp310-abi3-musllinux_1_2_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

warpllm-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

warpllm-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

warpllm-0.5.0-cp310-abi3-macosx_11_0_arm64.whl (2.3 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

warpllm-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file warpllm-0.5.0.tar.gz.

File metadata

  • Download URL: warpllm-0.5.0.tar.gz
  • Upload date:
  • Size: 435.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.5.0.tar.gz
Algorithm Hash digest
SHA256 ab8b58a547a19e7e2d361dab7b78ae175990ad03edf4cbad67f6544e6a0d7c22
MD5 824eba47be95ae6345f3e553f52fa354
BLAKE2b-256 6361e94469cdc6dd48385077e7fcb1806600f38183313dbabc43db33b609a8f0

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0.tar.gz:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-win_arm64.whl.

File metadata

  • Download URL: warpllm-0.5.0-cp310-abi3-win_arm64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.10+, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 7933d28f6ce0853ea9af03c20670503edc3bc37ea3df7b047444940ef63f19b3
MD5 1869eb1361361b195f1cf47c245278e9
BLAKE2b-256 b36f7b32992d4ac190e0a3bd63c918532386b41b61f6ff8bd30e8e76f7557ab9

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-win_arm64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: warpllm-0.5.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.5 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 c55e2eab1aa70a84a75c27fff12bcbfcce77d0d8b2127d3636c00e0d61995ef0
MD5 670110f7562e336eb6ae4fb52b7655ea
BLAKE2b-256 065c4db4149d29b138bf40cac5e5acc950c76fb1edc61dce0a0817937a7dd9ba

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-win_amd64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 5762acf4114af7bb2630091495c82bfdd50d3376db64106ed692cfa77bc21bc2
MD5 969c7cf504deae796e22672c429119b7
BLAKE2b-256 7b3ed720cc98a1fd2812eb9ba3cfd762abfc3cdcc5f8ecce4447ef3f202525cf

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 a886e88bace43cb960788ddb0f15bde33a5683419fc17f79abd18a653847c3f4
MD5 8a444831944029292c12b9ca666f0e54
BLAKE2b-256 8b5609a936aaa0ef3d52c8efec505ae924cad39b90c6ef18eb313c1512321860

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 34aa03b4803b3f9043cdd7a018e413ccae253fce81e2bebc42f0a323007027cc
MD5 3eb337210ab36e8a63d696f5c6abd1f1
BLAKE2b-256 b3e3539e874e1b81f525259f0c6ebb8e36e0d3f96edc3dfd77ed4c2a1a971c12

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ead5bf02a8345dc598693580f42be012f4231dfcb008b02294d0b8bc858617b7
MD5 0def85f983e5bdd1e5aa90d31c36c245
BLAKE2b-256 c4d9531c2e4b543d8bfaa8c53340e2ccd456ab3da53115304333716182bb6f24

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 61fe6c092c3055353bad5bd4454e64dc66f6f32a96488a20b3d66c92aefd1173
MD5 1dbe3c99ffb5c28e925432c51572d326
BLAKE2b-256 2387d9aa539c564246bb0a226f41f855c4c0d1ee94cd32c9103bd0a3f83d5494

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 3401e0eca4affe7222dd42b95afd4fd3dcc606ca75271af36328a6629c6c9fa1
MD5 5d8e8e5a12a0539fdf06c539efda139e
BLAKE2b-256 b76ad86e26a5be0b5865dc09f847a64cc233a05b6e65c7034980730c158ff3cb

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.5.0 This release

9 files

0.4.0

9 files

0.3.1

9 files

0.3.0

9 files

0.2.0

9 files

0.1.4

9 files

0.1.3

9 files

0.1.2

9 files

0.1.0

9 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page