LitServe: Deploy AI models Lightning fast ⚡

High-throughput serving engine for AI models.
Friendly interface. Enterprise scale.

LitServe is an engine for scalable AI model deployment built on FastAPI. Features like batching, streaming, and GPU autoscaling eliminate the need to rebuild a FastAPI server for each model.

✅ Batching       ✅ Streaming          ✅ Auto-GPU, multi-GPU   
✅ Multi-modal    ✅ PyTorch/JAX/TF     ✅ Full control          
✅ Auth           ✅ Built on Fast API  ✅ Custom specs (Open AI)

cpu-tests

Quick start • Lightning AI • Examples • Deploy • Features • Benchmarks • Docs

Performance

Our benchmarks show that LitServe (built on FastAPI) handles more simultaneous requests than FastAPI and TorchServe (higher is better).

Reproduce the full benchmarks here.

These results are for image and text classification ML tasks. The performance relationships hold for other ML tasks (embedding, LLM serving, audio, segmentation, object detection, summarization etc...).

💡 Note on LLM serving: For high-performance LLM serving (like Ollama/VLLM), use LitGPT or build your custom VLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.

Featured examples

Use LitServe to deploy any type of model or AI service (embeddings, LLMs, vision, audio, multi-modal, etc).

Featured examples

Toy model:  Hello world
LLMs:       Llama 3 (8B), LLM Proxy server
NLP:        Hugging face, BERT
Multimodal: OpenAI Clip
Audio:      Whisper, AudioCraft, StableAudio
Vision:     Stable diffusion 2
Speech:     Text-speech (XTTS V2)

Key features

✅ Serve all models: LLMs, vision, etc
✅ All frameworks:   PyTorch/Jax/sklearn/..
✅ Dev friendly:     build AI, not infra
✅ Easy interface:   no abstractions
✅ Enterprise scale: scale huge models
✅ Auto GPU scaling: zero code changes
✅ Self host:        or run on Studios

Quick start

Install LitServe via pip (or advanced installs):

pip install litserve

Define a server

Here's a hello world example (explore real examples):

# server.py
import litserve as ls

# STEP 1: DEFINE A MODEL API
class SimpleLitAPI(ls.LitAPI):
    # Called once at startup. Setup models, DB connections, etc...
    def setup(self, device):
        self.model = lambda x: x**2  

    # Convert the request payload to model input.
    def decode_request(self, request):
        return request["input"] 

    # Run inference on the the model, return the output.
    def predict(self, x):
        return self.model(x) 

    # Convert the model output to a response payload.
    def encode_response(self, output):
        return {"output": output} 

# STEP 2: START THE SERVER
if __name__ == "__main__":
    api = SimpleLitAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

Now run the server via the command-line

python server.py

LitAPI class gives full control and hackability.
LitServer handles optimizations like batching, auto-GPU scaling, etc...

Query the server

Use the automatically generated LitServe client or write your own:

Option A - Use generated client:           

python client.py

Option B - Custom client example:          

import requests
response = requests.post(
    "http://127.0.0.1:8000/predict",
    json={"input": 4.0}
)

Deployment options

Self-manage LitServe deployments (just run it on any machine!), or deploy with one click on Lightning AI.

LitServe is developed by Lightning AI which provides infrastructure for deploying AI models.

Feature	Self Managed	Fully Managed on Studios
Deployment	✅ Do it yourself deployment	✅ One-button cloud deploy
Load balancing	❌	✅
Autoscaling	❌	✅
Scale to zero	❌	✅
Multi-machine inference	❌	✅
Authentication	❌	✅
Own VPC	❌	✅
AWS, GCP	❌	✅
Use your own cloud commits	❌	✅

Features

LitServe supports multiple advanced state-of-the-art features.

✅ All model types: LLMs, vision, time series, etc...
✅ Auto-GPU scaling
✅ Authentication
✅ Autoscaling
✅ Batching
✅ Streaming
✅ All ML frameworks: PyTorch, Jax, Tensorflow, Hugging Face...
✅ Open AI spec

10+ features...

Note: Our goal is not to jump on every hype train, but instead support features that scale under the most demanding enterprise deployments.

Community

LitServe is a community project accepting contributions - Let's make the world's most advanced AI inference engine.

💬 Get help on Discord
📋 License: Apache 2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

litserve-0.1.5.tar.gz (29.9 kB view details)

Uploaded Aug 2, 2024 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

litserve-0.1.5-py3-none-any.whl (30.4 kB view details)

Uploaded Aug 2, 2024 Python 3

File details

Details for the file litserve-0.1.5.tar.gz.

File metadata

Download URL: litserve-0.1.5.tar.gz
Upload date: Aug 2, 2024
Size: 29.9 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for litserve-0.1.5.tar.gz
Algorithm	Hash digest
SHA256	`ad8ec0f6b10207e9a14e58240e6c6e330db3e00c32e778843941e4fd916d4c82`
MD5	`eef48898fb27f6f0b30ae39454164bae`
BLAKE2b-256	`6f4da3903501d07bf3202cb4b9059569820b05420657c4106a9510c394665afd`

See more details on using hashes here.

File details

Details for the file litserve-0.1.5-py3-none-any.whl.

File metadata

Download URL: litserve-0.1.5-py3-none-any.whl
Upload date: Aug 2, 2024
Size: 30.4 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for litserve-0.1.5-py3-none-any.whl
Algorithm	Hash digest
SHA256	`c1c6b34275ee03d552df166dd49c185b6775e363bbab0084c4df5ac6f2f85c82`
MD5	`268ef16c1e083a5a5b1055d97ac4a6c2`
BLAKE2b-256	`28c97e6066ced9728cfc5823737a451304a74f29e25979a1b5485942819a5667`