Skip to main content

warpllm

A warp-speed, robust AI gateway written for rust, node, and python applications - built for planet scale by the community.

Discord Reddit

CI crates.io PyPI npm License

Quickstart

pip install warpllm              # python
npm install @warpllm/warpllm     # node
cargo add warpllm                # rust
export OPENAI_API_KEY=sk-...

Python

from warpllm import WarpLLM

client = WarpLLM()

completion = client.chat_completions({
    "model": "openai/gpt-5-nano",
    "messages": [{"role": "user", "content": "Hello!"}],
})

print(completion["choices"][0]["message"]["content"])

Node

import { WarpLLM } from '@warpllm/warpllm'

const client = new WarpLLM()

const completion = await client.chatCompletions({
  model: 'openai/gpt-5-nano',
  messages: [{ role: 'user', content: 'Hello!' }],
})

console.log(completion.choices[0].message.content)

Rustchat_completions is async and warpllm ships no runtime, so bring your own: cargo add tokio --features macros,rt-multi-thread.

use warpllm::{ChatCompletionRequestMessage, Client, ClientConfig, CreateChatCompletionRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new(ClientConfig::default())?;

    let completion = client
        .chat_completions(CreateChatCompletionRequest {
            model: "openai/gpt-5-nano".to_string(),
            messages: vec![ChatCompletionRequestMessage {
                role: "user".to_string(),
                content: "Hello!".to_string(),
                ..Default::default()
            }],
            ..Default::default()
        })
        .await?;

    let content = completion.choices[0].message.content.as_deref();
    println!("{}", content.unwrap_or_default());
    Ok(())
}

Switching providers is a string change

Keys are read from the environment when the client is built, so export the one the provider needs and change the model string. Nothing else moves.

Model string Key it needs
openai/gpt-5-nano OPENAI_API_KEY
deepseek/deepseek-v4-flash DEEPSEEK_API_KEY
kimi/kimi-k3 MOONSHOT_API_KEY
opencode/glm-5.2 OPENCODE_API_KEY
openrouter/anthropic/claude-sonnet-4 OPENROUTER_API_KEY

The provider/ prefix is required. warpllm matches the whole string against its roster, so a bare gpt-5-nano — or any name it doesn't know — is an error rather than a guess at an upstream default.

Narrowing a client to the providers it serves

By default a client serves the whole roster and reads every provider's variable. Declare the ones you mean and it reads no others, routes to no others, and takes a key directly for the callers who keep theirs somewhere the environment can't reach:

WarpLLM(providers={"openai": {}, "deepseek": {"api_key": "sk-..."}})
new WarpLLM({ providers: { openai: {}, deepseek: { apiKey: 'sk-...' } } })
ClientConfig {
    providers: Some(BTreeMap::from([
        ("openai".into(), ProviderConfig::default()),
        ("deepseek".into(), ProviderConfig { api_key: Some(key) }),
    ])),
    ..Default::default()
}

An empty entry means "serve this one, key from the environment". A request for a model under a provider you didn't declare is refused before any upstream call, and a provider name the roster doesn't hold fails when the client is built.

Runnable versions of all three, with comments, are in examples/.

Mission

This project is to lay out the most resilient open source productionization layer for AI-deployments. Designed for you if you want:

  1. To work with multiple AI providers or your own models.
  2. To keep your AI services up and running with 0 downtime.
  3. Speed (minimal overhead latency).
  4. A granular view of your metrics (uptime, P95 latency, costs, etc).
  5. Control over:
    1. Where your data goes.
    2. Your AI budget across providers.

Status

[!IMPORTANT] The published packages are 0.4.0, which adds the OpenCode Zen provider and lets a client declare the providers it serves. It is source-breaking for Rust only: ClientConfig gained a field, so an exhaustive struct literal no longer compiles — add providers: None, or switch to ..Default::default(). Python and TypeScript are purely additive. See the changelog before upgrading from 0.3.x.

The OpenAI-compatible HTTP gateway has landed on main but is not released yet.

Released (0.4.0) On main
OpenAI chat completions, non-streaming Yes Yes
provider/model routing strings Provider registry Provider registry
DeepSeek, OpenRouter Yes Yes
Kimi Yes Yes
OpenCode Zen Yes Yes
Declaring the providers a client serves Yes Yes
OpenAI-compatible HTTP gateway Unreleased
Streaming Yes Yes
Failover, load balancing, caching, metrics

Unlisted models are rejected rather than guessed at, so routing a name warpllm doesn't know is an error, not a surprise upstream bill.

Layers

  1. An SDK - provide a request and we translate it to work with different providers and models out of box.
  2. [Unreleased] A proxy - run a self-hosted proxy that speaks the OpenAI API:
    1. [Coming Soon] Failover - define multiple models to handle outages / errors
    2. [Coming Soon] Load Balancing - define a % of requests to be handled per model
    3. [Coming Soon] Prompt Response Caching - define a TTL and avoid paying twice for the same prompt

Key focus points

  1. Native SDK support - Written once in rust, compiled for maximum performance, available for rust/typescript/python.
  2. Self hostable - Avoid vendor lock-in (e.g. from cloud provider or model provider), or data leaving your infra.
  3. Warp-speed execution - What we named ourselves after. Machine level code, faster than a typescript or python native library.
  4. Compact file size - Pre-compiled into binary format, not verbose text files.

Roadmap

The roadmap lives in GitHub issues — one issue per item, so direction is discussed where the work happens. Add a comment if you see something missing, or if something there matters enough to you that it should move up.

Contributing

We're excited to have you join us. See the contribution guide for how to get started.

A big thank you to the contributors below who have helped build this AI gateway to this point!

License

The warpllm core is open source under the Apache License 2.0.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

warpllm-0.4.0.tar.gz (357.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

warpllm-0.4.0-cp310-abi3-win_arm64.whl (2.3 MB view details)

Uploaded CPython 3.10+Windows ARM64

warpllm-0.4.0-cp310-abi3-win_amd64.whl (2.5 MB view details)

Uploaded CPython 3.10+Windows x86-64

warpllm-0.4.0-cp310-abi3-musllinux_1_2_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

warpllm-0.4.0-cp310-abi3-musllinux_1_2_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

warpllm-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

warpllm-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

warpllm-0.4.0-cp310-abi3-macosx_11_0_arm64.whl (2.2 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

warpllm-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file warpllm-0.4.0.tar.gz.

File metadata

  • Download URL: warpllm-0.4.0.tar.gz
  • Upload date:
  • Size: 357.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.4.0.tar.gz
Algorithm Hash digest
SHA256 953d7f7ccb8845a78aafa780e242df65aee91a08216e35f144424c32513cb9ea
MD5 b8ea66fcf81d0843fd52a0487bd0956b
BLAKE2b-256 21a2916a4f1ee1d5b7250b1269377e9e45261427869aca9d397bf9bfa2e977ff

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0.tar.gz:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-win_arm64.whl.

File metadata

  • Download URL: warpllm-0.4.0-cp310-abi3-win_arm64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10+, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 350eb15ed56732d52b2b338bde5b9b9a181fe79ae5ab284a1ac0a4d21e85674e
MD5 84015bb74e3416afa63e111613c95c5e
BLAKE2b-256 51ce8cd3c75641ebf7bdbd69090581c34d5170cd72a2080309faf73168ddeec2

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-win_arm64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: warpllm-0.4.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.5 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 08c4b1a8890ff9617d5e40f16f15dcf4f29e88f4a6370484c13f64b8dff8de8c
MD5 84e9a5b2a2a9a8ac8ec91c388d46647d
BLAKE2b-256 8cff68a2a7ca64fea74cb26b67a9aba9d9dacae50ba689f9f78a146ce51a0c2b

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-win_amd64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 a92f3ee8520e3c262ebda40fb80f2353fc53662bb34dbb11fca003fcdf89823b
MD5 e6dbacbc7bbb074aa089fc86d6930c86
BLAKE2b-256 f5c596d9fc1a1452a1b3ef24d7970a2a2cd7121103e1de1cb67cf934f8300ced

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 d5dc4e65225c4cea1e6c0c83d74fe034e588290d6b57290979da8a1bed642828
MD5 f876ca781af712dfb2d985165008b5cf
BLAKE2b-256 1994b5ba758490446a8f0cb8a35912217177246790abf0fd23f504844de23b8f

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 72ff68d84ad5d9b8106d9177def46c6bcfb17b5b0d2aeda312b815619b9dfafa
MD5 b65e2b76b5aa8b1701af5e642f4f8333
BLAKE2b-256 4d04e61c439814ec9b874e2cf690a3c5ee862c9c1024266141efe4393cb70373

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8b369361410f1abccad19a90df67ffeae8fb0eaa3abe7fb518b87b82b0afd824
MD5 a4487e5891a587bc93d8cb21530fc192
BLAKE2b-256 94c4afd4c704ea4e46fd14a466e2d2c7d7776765b6842008d3c87286e34ebd3f

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8a8ca55f0847e9593fd8cbd998fee75a1dd3df7c26d92a88eb3467fe24bf70e9
MD5 87b0dddee6edd08b2ad4a15bf8078591
BLAKE2b-256 699397b7544e2780aa4778ab04bb87b94189c675f0ec03ee60ccd42d7c7260de

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file warpllm-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for warpllm-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 45e9bc8ab8bcf7d703f90f0833294481e327b2c13510dbc03aafff4404ead97c
MD5 89ef08a194532f2931d301c5a67a7122
BLAKE2b-256 320007a037d9b52035ff19e126eec144d52bf2d17180430a93c18a120ebb64c4

See more details on using hashes here.

Provenance

The following attestation bundles were made for warpllm-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release-python.yml on warpllm/warpllm

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.5.0

9 files

This release

0.4.0 This release

9 files

0.3.1

9 files

0.3.0

9 files

0.2.0

9 files

0.1.4

9 files

0.1.3

9 files

0.1.2

9 files

0.1.0

9 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page