Deploy agents, RAG, models, pipelines and more.
Without learning MLOps.

 

Most AI inference tools are built around single-model APIs with rigid abstractions. They lock you into serving one model per server, with no way to customize internals like batching, caching, or kernels. This makes it hard to build full systems like RAG or agents without stitching together multiple services. The result is complex MLOps orchestration, slower iteration, and bloated infrastructure.

LitServe flips this paradigm: Write full AI pipelines, not just models, in clean, extensible Python. Built on FastAPI but optimized for AI workloads, LitServe supports multi-model serving, streaming, batching, and custom logic - all from a single server. Deploy in one click with autoscaling, monitoring, and zero infrastructure overhead. Or run it self-hosted with full control and no lock-in.

LitServe is at least 2x faster than plain FastAPI due to AI-specific multi-worker handling.

✅ (2x)+ faster serving  ✅ Easy to use               ✅ LLMs, non LLMs and more
✅ Bring your own model  ✅ PyTorch/JAX/TF/...        ✅ Built on FastAPI       
✅ GPU autoscaling       ✅ Batching, Streaming       ✅ Self-host or ⚡️ managed
✅ Inference pipeline    ✅ Integrate with vLLM, etc  ✅ Serverless             
   

Quick start

Install LitServe via pip (more options):

pip install litserve

Define a server

This toy example with 2 models (inference pipeline) shows LitServe's flexibility (see real examples):

import litserve as ls

# define the api to include any number of models, dbs, etc...
class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        self.model1 = lambda x: x**2
        self.model2 = lambda x: x**3

    def decode_request(self, request):
        # get inputs to /predict
        return request["input"]

    def predict(self, x):
        # perform calculations using both models
        a = self.model1(x)
        b = self.model2(x)
        c = a + b
        return {"output": c}

    def encode_response(self, output):
        # package outputs from /predict
        return {"output": output} 

if __name__ == "__main__":
    # 12+ features like batching, streaming, etc...
    server = ls.LitServer(SimpleLitAPI(max_batch_size=1), accelerator="auto")
    server.run(port=8000)

Now deploy for free to Lightning cloud (or self host anywhere):

# Deploy for free with autoscaling, monitoring, etc...
lightning deploy server.py --cloud

# Or run locally (self host anywhere)
lightning deploy server.py
# python server.py

Test the server

Simulate an http request (run this on any terminal):

curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"input": 4.0}'

LLM serving

LitServe isn’t just for LLMs like vLLM or Ollama; it serves any AI model with full control over internals (learn more).
For easy LLM serving, integrate vLLM with LitServe, or use LitGPT (built on LitServe).

litgpt serve microsoft/phi-2

Summary

• LitAPI lets you easily build complex AI systems with one or more models (docs).
• Use the setup method for one-time tasks like connecting models, DBs, and loading data (docs).
• LitServer handles optimizations like batching, GPU autoscaling, streaming, etc... (docs).
• Self host on your machines or create a fully managed deployment with Lightning (learn more).

Learn how to make this server 200x faster.

Featured examples

Here are examples of inference pipelines for common model types and use cases.

Toy model:      Hello world
LLMs:           Llama 3.2, LLM Proxy server, Agent with tool use
RAG:            vLLM RAG (Llama 3.2), RAG API (LlamaIndex)
NLP:            Hugging face, BERT, Text embedding API
Multimodal:     OpenAI Clip, MiniCPM, Phi-3.5 Vision Instruct, Qwen2-VL, Pixtral
Audio:          Whisper, AudioCraft, StableAudio, Noise cancellation (DeepFilterNet)
Vision:         Stable diffusion 2, AuraFlow, Flux, Image Super Resolution (Aura SR),
                Background Removal, Control Stable Diffusion (ControlNet)
Speech:         Text-speech (XTTS V2), Parler-TTS
Classical ML:   Random forest, XGBoost
Miscellaneous:  Media conversion API (ffmpeg), PyTorch + TensorFlow in one API, LLM proxy server

Browse 100+ community-built templates

Hosting options

Self host LitServe anywhere or deploy to your favorite cloud via Lightning AI.

https://github.com/user-attachments/assets/ff83dab9-0c9f-4453-8dcb-fb9526726344

Self-hosting is ideal for hackers, students, and DIY developers while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.

Note: Lightning offers a generous free tier for developers.

To host on Lightning AI, simply run the command, login and choose the cloud of your choice.

lightning deploy server.py --cloud

Features

Performance

LitServe is designed for AI workloads. Specialized multi-worker handling delivers a minimum 2x speedup over FastAPI.

Additional features like batching and GPU autoscaling can drive performance well beyond 2x, scaling efficiently to handle more simultaneous requests than FastAPI and TorchServe.

Reproduce the full benchmarks here (higher is better).

These results are for image and text classification ML tasks. The performance relationships hold for other ML tasks (embedding, LLM serving, audio, segmentation, object detection, summarization etc...).

💡 Note on LLM serving: For high-performance LLM serving (like Ollama/vLLM), integrate vLLM with LitServe, use LitGPT, or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.

Community

LitServe is a community project accepting contributions - Let's make the world's most advanced AI inference engine.

💬 Get help on Discord
📋 License: Apache 2.0