inferweave 0.2.0

Unified AI inference deployment SDK across cloud GPU providers

Unified AI inference deployment SDK across cloud GPU providers.
Choose your model or workload (Audio, LLM, Image, Video) — InferWeave provisions the GPU infrastructure, sets up the container environment, manages CUDA/VRAM requirements, and exposes a ready-to-use endpoint.

🌟 The Vision

Deploying open-weight AI models into production is notoriously complex:

• Different hardware specs (VRAM sizing, tensor parallelism, compute capability).
• Fragmented container runtimes (vLLM, SGLang, ComfyUI, custom FastAPI/PyTorch wrappers).
• Provider lock-in and disparate provisioning APIs (RunPod, Lambda, AWS, GCP, Modal, etc.).

InferWeave hides all infrastructural complexity behind a unified Python interface:

Configure account and Proxy Token credentials using the TTS tutorial, then:

"""Run with python examples/tts.py after configuring Modal and proxy credentials."""

import asyncio
from pathlib import Path

from inferweave import InferWeave


async def main() -> None:
    weave = InferWeave()
    deployment = None
    try:
        deployment = await weave.deploy(
            model="fish-s2-pro",
            provider="modal",
            custom_args={"cleanup_on_failure": True},
        )
        audio = await deployment.synthesize("Hello from InferWeave", format="wav")
        Path("speech.wav").write_bytes(audio)
    finally:
        try:
            if deployment is not None:
                await deployment.stop()
        finally:
            await weave.close()


if __name__ == "__main__":
    asyncio.run(main())

InferWeave automatically handles model profile detection, VRAM budgeting, container runtime templates, healthcheck polling, and endpoint proxying.

Inference, recovery and lifecycle

await weave.attach(id) rebuilds a handle after restart; await weave.find(model=..., provider=...) reuses a unique active deployment from SQLite. Ambiguity raises AmbiguousDeploymentError. Mount the database directory in persistent services.

Modal endpoints are protected by default. SDK credentials provision resources; separate MODAL_PROXY_TOKEN_ID/MODAL_PROXY_TOKEN_SECRET authenticate probes/inference. Scale-to-zero releases idle GPU containers while keeping the app reusable; destroy_after_idle_mins stops the entire app. close() releases local resources.

Documentation and examples

Lightning AI uses the same deployment, inference and recovery API:

weave = InferWeave()
deployment = await weave.deploy("fish-s2-pro", provider="lightning")
audio = await deployment.synthesize("Hello from Lightning AI")
await deployment.stop()
await weave.close()

Install pip install "inferweave[lightning]" and configure LIGHTNING_USER_ID, LIGHTNING_API_KEY and LIGHTNING_TEAMSPACE=owner/teamspace locally. The user API key also authenticates inference; scoped keys are unsupported. Lightning creates one container Deployment with no Studio; stop deletes it. Read the Lightning guide for first-pull readiness, autoscaling, restart recovery and opt-in paid tests. The verified SDK is lightning-sdk==2026.10.1 (Python 3.11/3.12). Its Click requirement conflicts with SkyPilot, so use a separate environment. inferweave[all] retains SkyPilot, Modal and workers; it excludes Lightning.

• Use Lightning AI
• First TTS on Modal
• Generate images
• Recover deployments
• Secure endpoints
• Handle cold starts
• Long-lived services
• API, configuration, models
• Architecture, lifecycle and cost
• Runnable examples, roadmap, changelog

🏗️ Architecture

               ┌────────────────────────┐
               │    Your Application    │
               └───────────┬────────────┘
                           │
               ┌───────────▼────────────┐
               │     InferWeave SDK     │
               │ (Registry, Router, API)│
               └───────────┬────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌───────────────┐                       ┌───────────────┐
│ Runtime Layer │                       │ Compute Layer │
├───────────────┤                       ├───────────────┤
│ • Audio (TTS) │                       │ • SkyPilot:   │
│ • LLM (vLLM)  │                       │   RunPod/Vast │
│ • Image (FLUX)│                       │   AWS/GCP/etc.│
│ • Video (WAN) │                       │ • Modal       │
│ • Custom      │                       │ • Cloud GPUs  │
└───────────────┘                       └───────────────┘

📦 Installation

InferWeave provides a modular installation model so you only install dependencies for the clouds you actually use.

1. Base Installation (Core SDK & CLI)

Installs the unified client, schema definitions, model registry, and CLI tools:

pip install inferweave

2. Provider-Specific Extras

Install drivers for the clouds you plan to target:

# Individual cloud providers (via SkyPilot engine)
pip install "inferweave[runpod]"
pip install "inferweave[aws]"
pip install "inferweave[gcp]"
pip install "inferweave[azure]"
pip install "inferweave[lambda]"
pip install "inferweave[nebius]"
pip install "inferweave[kubernetes]"
pip install "inferweave[vast]"

# Serverless compute engine (verified with modal>=1.6,<1.7)
pip install "inferweave[modal]"

# Inference worker runtime (FastAPI / Uvicorn stack)
pip install "inferweave[workers]"

3. Multi-Cloud Bundles

# All cloud providers supported by SkyPilot
pip install "inferweave[clouds]"

# SkyPilot cloud engines + Modal + workers runtime (Lightning uses a separate environment)
pip install "inferweave[all]"

⚡ CLI Quickstart

InferWeave includes a high-performance command-line interface for deploying and inspecting models across GPU providers:

# 1. Deploy models directly to cloud GPUs
inferweave deploy fish-s2-pro --provider modal
inferweave deploy meta-llama/Meta-Llama-3-8B-Instruct --provider runpod --gpu A100

# 2. Dry-run deployment to preview configuration and hardware budgets
inferweave deploy fish-s2-pro --provider modal --dry-run

# 3. Explore supported models and cloud providers
inferweave models
inferweave models --workload audio
inferweave providers

# 4. Inspect, monitor, and stop deployments across processes
inferweave list
inferweave status <deployment-id>
inferweave stop <deployment-id>

💻 Platform & OS Support Guidelines

⚠️ Important Note for Windows Users

SkyPilot’s underlying execution engine relies on POSIX system primitives (termios, resource) that are not present in native Windows.

• If you are on Windows: We recommend running InferWeave inside WSL2 (Windows Subsystem for Linux), where all multi-cloud provisioning features work seamlessly.
• InferWeave Graceful Fallback: InferWeave utilizes lazy loading for SkyPilot modules. If invoked directly on native Windows, InferWeave will safely guide you to WSL2 or allow targeting native-supported backends (such as Modal or remote runners) without crashing during import.

🛠️ Contributing & Development Setup

InferWeave uses uv for lightning-fast dependency management.

# Clone the repository
git clone https://github.com/AmaLS367/InferWeave.git
cd InferWeave

# Install extras needed for mocked tests and dev tools
uv sync --extra modal --extra workers --group dev

# Run linting and type checking
uv run ruff check .
uv run mypy src
uv run pytest -m "not integration" --strict-markers -ra
uv run python examples/simulate.py

Live integration tests (opt-in, spend real money)

Tests marked integration talk to real clouds. They are excluded from CI and skip cleanly unless explicitly enabled and configured:

For local testing, copy .env.example to .env and fill in the four Modal credentials. .env is ignored by Git. The library does not load it automatically; uv can load it for the test process:

uv run --env-file .env --extra modal pytest -m integration tests/test_modal_integration.py --strict-markers -ra

This test checks protected access, TTS output, recovery with a fresh SDK and deployment cleanup. It provisions a real GPU and incurs charges.

The RunPod test also reads optional INFERWEAVE_RUNPOD_GPU (default L4), INFERWEAVE_RUNPOD_MODEL (default facebook/opt-125m), INFERWEAVE_RUNPOD_READY_TIMEOUT_SECS, INFERWEAVE_RUNPOD_AUTOSTOP_MINS and INFERWEAVE_RUNPOD_ALLOW_SPOT; see the module docstring.

INFERWEAVE_RUNPOD_INTEGRATION=1 RUNPOD_API_KEY=... \
  uv run --extra runpod pytest -m integration tests/test_runpod_integration.py -s

🚀 Release Process

Publishing to PyPI is automated via GitHub Actions using PyPI Trusted Publishing (OIDC):

1. Ensure the CI suite is green on master.
2. Set the package version in src/inferweave/__init__.py.
3. Create and push a Git tag: vX.Y.Z.
4. Publish a GitHub Release from that tag.
5. GitHub Actions (publish.yml) verifies artifact metadata, validates that the release tag matches inferweave.__version__, and publishes the artifacts to PyPI via Trusted Publishing.

📄 License

InferWeave's source code is licensed under the Apache License 2.0. See LICENSE for details.

Third-party licenses

The Apache-2.0 license covers InferWeave's own source code only. It does not cover anything InferWeave deploys or references:

• Model weights (including those named by the built-in model profiles) keep their upstream licenses.
• Container images and third-party runtimes (vLLM, Fish Speech, PyTorch, and the Python packages installed into them) keep their upstream licenses.
• You are responsible for reviewing and complying with the upstream terms of every model, image and runtime you deploy. This is not legal advice.

Fish Audio S2 Pro (fish-s2-pro): the fishaudio/s2-pro weights are published under Fish Audio's own license (fish-audio-research-license on Hugging Face), not Apache-2.0. The built-in S2 runtime is beta: it uses the upstream v2.0.0-beta pre-release of fishaudio/fish-speech. Review the upstream model and runtime licenses before any production or commercial use.