Quickstart

pip install warpllm              # python
npm install @warpllm/warpllm     # node
cargo add warpllm                # rust

export OPENAI_API_KEY=sk-...

Python

from warpllm import WarpLLM

client = WarpLLM()

completion = client.chat_completions({
    "model": "openai/gpt-5-nano",
    "messages": [{"role": "user", "content": "Hello!"}],
})

print(completion["choices"][0]["message"]["content"])

Node

import { WarpLLM } from '@warpllm/warpllm'

const client = new WarpLLM()

const completion = await client.chatCompletions({
  model: 'openai/gpt-5-nano',
  messages: [{ role: 'user', content: 'Hello!' }],
})

console.log(completion.choices[0].message.content)

Rust — chat_completions is async and warpllm ships no runtime, so bring your own: cargo add tokio --features macros,rt-multi-thread.

use warpllm::{ChatCompletionRequestMessage, Client, ClientConfig, CreateChatCompletionRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new(ClientConfig::default())?;

    let completion = client
        .chat_completions(CreateChatCompletionRequest {
            model: "openai/gpt-5-nano".to_string(),
            messages: vec![ChatCompletionRequestMessage {
                role: "user".to_string(),
                content: "Hello!".to_string(),
                ..Default::default()
            }],
            ..Default::default()
        })
        .await?;

    let content = completion.choices[0].message.content.as_deref();
    println!("{}", content.unwrap_or_default());
    Ok(())
}

Switching providers is a string change

Keys are read from the environment when the client is built, so export the one the provider needs and change the model string. Nothing else moves.

The provider/ prefix is required. warpllm matches the whole string against its roster, so a bare gpt-5-nano — or any name it doesn't know — is an error rather than a guess at an upstream default.

Runnable versions of all three, with comments, are in examples/.

Mission

This project is to lay out the most resilient open source productionization layer for AI-deployments. Designed for you if you want:

1. To work with multiple AI providers or your own models.
2. To keep your AI services up and running with 0 downtime.
3. Speed (minimal overhead latency).
4. A granular view of your metrics (uptime, P95 latency, costs, etc).
5. Control over:
   1. Where your data goes.
   2. Your AI budget across providers.

Status

[!IMPORTANT] The published packages are 0.3.1, which adds streaming — from the Rust core, both SDKs, and the HTTP gateway — alongside the Kimi provider and the rest of OpenAI's chat-completion roster. It is a breaking release: response fields that are optional and nullable now tell an absent key from an explicit null, so their type changed in all three languages. See the changelog before upgrading from 0.2.x.

The OpenAI-compatible HTTP gateway has landed on main but is not released yet.

Unlisted models are rejected rather than guessed at, so routing a name warpllm doesn't know is an error, not a surprise upstream bill.

Layers

1. An SDK - provide a request and we translate it to work with different providers and models out of box.
2. [Unreleased] A proxy - run a self-hosted proxy that speaks the OpenAI API:
   1. [Coming Soon] Failover - define multiple models to handle outages / errors
   2. [Coming Soon] Load Balancing - define a % of requests to be handled per model
   3. [Coming Soon] Prompt Response Caching - define a TTL and avoid paying twice for the same prompt

Key focus points

1. Native SDK support - Written once in rust, compiled for maximum performance, available for rust/typescript/python.
2. Self hostable - Avoid vendor lock-in (e.g. from cloud provider or model provider), or data leaving your infra.
3. Warp-speed execution - What we named ourselves after. Machine level code, faster than a typescript or python native library.
4. Compact file size - Pre-compiled into binary format, not verbose text files.

Roadmap

The roadmap lives in GitHub issues — one issue per item, so direction is discussed where the work happens. Add a comment if you see something missing, or if something there matters enough to you that it should move up.

Contributing

We're excited to have you join us. See the contribution guide for how to get started.

A big thank you to the contributors below who have helped build this AI gateway to this point!

License

The warpllm core is open source under the Apache License 2.0.