Quickstart

pip install warpllm              # python
npm install @warpllm/warpllm     # node
cargo add warpllm                # rust

export OPENAI_API_KEY=sk-...

Python

from warpllm import WarpLLM

client = WarpLLM()

completion = client.chat_completions({
    "model": "openai/gpt-5-nano",
    "messages": [{"role": "user", "content": "Hello!"}],
})

print(completion["choices"][0]["message"]["content"])

Node

import { WarpLLM } from '@warpllm/warpllm'

const client = new WarpLLM()

const completion = await client.chatCompletions({
  model: 'openai/gpt-5-nano',
  messages: [{ role: 'user', content: 'Hello!' }],
})

console.log(completion.choices[0].message.content)

Rust — chat_completions is async and warpllm ships no runtime, so bring your own: cargo add tokio --features macros,rt-multi-thread.

use warpllm::{ChatCompletionRequestMessage, Client, ClientConfig, CreateChatCompletionRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new(ClientConfig::default())?;

    let completion = client
        .chat_completions(CreateChatCompletionRequest {
            model: "openai/gpt-5-nano".to_string(),
            messages: vec![ChatCompletionRequestMessage {
                role: "user".to_string(),
                content: "Hello!".to_string(),
                ..Default::default()
            }],
            ..Default::default()
        })
        .await?;

    let content = completion.choices[0].message.content.as_deref();
    println!("{}", content.unwrap_or_default());
    Ok(())
}

Switching providers is a string change

Keys are read from the environment when the client is built, so export the one the provider needs and change the model string. Nothing else moves.

The provider/ prefix is required. warpllm matches the whole string against its roster, so a bare gpt-5-nano — or any name it doesn't know — is an error rather than a guess at an upstream default.

Narrowing a client to the providers it serves

By default a client serves the whole roster and reads every provider's variable. Declare the ones you mean and it reads no others, routes to no others, and takes a key directly for the callers who keep theirs somewhere the environment can't reach:

WarpLLM(providers={"openai": {}, "deepseek": {"api_key": "sk-..."}})

new WarpLLM({ providers: { openai: {}, deepseek: { apiKey: 'sk-...' } } })

ClientConfig {
    providers: Some(BTreeMap::from([
        ("openai".into(), ProviderConfig::default()),
        ("deepseek".into(), ProviderConfig { api_key: Some(key) }),
    ])),
    ..Default::default()
}

An empty entry means "serve this one, key from the environment". A request for a model under a provider you didn't declare is refused before any upstream call, and a provider name the roster doesn't hold fails when the client is built.

Runnable versions of all three, with comments, are in examples/.

Your own models

Anything that speaks the OpenAI API — vLLM, TGI, Ollama, llama.cpp — is a routing target. Describe it in a file and hand warpllm the path:

# ./warpllm.yaml
providers:
  local:
    base_url: "http://localhost:8000/v1"
    auth: none                       # the box is on a private network
    models:
      local/llama-3.3-70b:
        supported_apis:
          - {api: openai_compat_chat_completions}
          - {api: openai_compat_chat_completions_stream}

client = WarpLLM(specs_path="./warpllm.yaml")
client.chat_completions({"model": "local/llama-3.3-70b", "messages": [...]})

const client = new WarpLLM({ specsPath: './warpllm.yaml' })

let client = Client::new(ClientConfig {
    specs_path: Some("./warpllm.yaml".into()),
    ..Default::default()
})?;

warpllm-server --specs ./warpllm.yaml   # or WARPLLM_SPECS=./warpllm.yaml

Your file is merged over the built-in roster, so adding local/ leaves openai/ exactly where it was — the same client routes both. Reusing a built-in provider's name replaces that provider whole, and warpllm warns rather than shadowing it quietly. The warning goes through tracing, which warpllm-server surfaces and a Rust client does once it installs a subscriber; the Python and Node bindings install none yet, so there it goes nowhere. Same for the older warning about an environment with no provider keys in it.

auth: none is the line that matters for a private box: warpllm then sends no Authorization header at all. Omitting it means something else — that the roster records no way to authenticate this provider — so a forgotten env_api_key on a paid provider fails locally instead of leaving without a credential.

The file is read when the client is built, so a roster that can't be used is an error there, naming the path — not a request failing hours later. There is no wildcard: every model gets an entry, because supported_apis and capabilities are per model and a pattern would have to claim both on behalf of models nobody listed.

The schema is documented in full at the top of specs.yaml, and examples/warpllm.yaml is a worked one covering vLLM, Ollama, and a cluster that does want a key.

Mission

This project is to lay out the most resilient open source productionization layer for AI-deployments. Designed for you if you want:

1. To work with multiple AI providers or your own models.
2. To keep your AI services up and running with 0 downtime.
3. Speed (minimal overhead latency).
4. A granular view of your metrics (uptime, P95 latency, costs, etc).
5. Control over:
   1. Where your data goes.
   2. Your AI budget across providers.

Status

[!IMPORTANT] The published packages are 0.5.0, which lets a client bring its own roster file — so a self-hosted OpenAI-compatible server is a routable target without forking the crate — and adds weighted load balancing (Rust only) and Mistral. It is source-breaking for Rust only: ClientConfig gained a field, so an exhaustive struct literal no longer compiles — add specs_path: None, or switch to ..Default::default(). Python and TypeScript are purely additive. See the changelog before upgrading from 0.4.x.

The OpenAI-compatible HTTP gateway has landed on main but is not released yet.

Unlisted models are rejected rather than guessed at, so routing a name warpllm doesn't know is an error, not a surprise upstream bill.

Layers

1. An SDK - provide a request and we translate it to work with different providers and models out of box.
2. [Unreleased] A proxy - run a self-hosted proxy that speaks the OpenAI API:
   1. [Coming Soon] Failover - define multiple models to handle outages / errors
   2. [Coming Soon] Load Balancing - define a % of requests to be handled per model
   3. [Coming Soon] Prompt Response Caching - define a TTL and avoid paying twice for the same prompt

Key focus points

1. Native SDK support - Written once in rust, compiled for maximum performance, available for rust/typescript/python.
2. Self hostable - Avoid vendor lock-in (e.g. from cloud provider or model provider), or data leaving your infra.
3. Warp-speed execution - What we named ourselves after. Machine level code, faster than a typescript or python native library.
4. Compact file size - Pre-compiled into binary format, not verbose text files.

Roadmap

The roadmap lives in GitHub issues — one issue per item, so direction is discussed where the work happens. Add a comment if you see something missing, or if something there matters enough to you that it should move up.

Contributing

We're excited to have you join us. See the contribution guide for how to get started.

A big thank you to the contributors below who have helped build this AI gateway to this point!

License

The warpllm core is open source under the Apache License 2.0.