FlockServe

Open-source Sky Computing Inference Endpoint

Overview

FlockServe is an open-source library for deploying production-ready AI inference endpoints. Similar to closed-source commercial inference endpoints, FlockServe adds the capabilities of autoscaling, load balancing, and monitoring to an inference engine server, turning these into production-ready solutions for serving AI predictions at dynamic request rates and high volumes.

FlockServe uses SkyPilot as the node provisioner, taking SkyPilot task files for single inference engine servers and autoscales these based on request volume. Due to using SkyPilot, an inference endpoint developed with FlockServe will work natively across multiple clouds, and can be migrated between providers just by changing the SkyPilot configuration.

Any inference engine server can be used, and examples of SkyPilot task files are given for vLLM and TGI. Both the OpenAI and /generate APIs used by these are supported. FlockServe runs on FastAPI and uvicorn, so requests are processed fully asynchronously.

FlockServe has a modular design, and different solutions for autoscaling and load balancing can be used. The default option for autoscaling uses a running mean estimate of request queue lengths, which is effective for LLM autoscaling. The default option for load balancing uses Least Connection Load Balancing, which is also well suited for serving LLMs.

Features

• Scalability: Easily scale your inference endpoint based on the demand using cloud resources.
• Skypilot Integration: Leverages the power of skypilot for sky computing free from vendor lock-in
• Flexible Model Support: Supports any inference engine such as vLLM and TGI, and models supported by these.
• RESTful API: Simple and intuitive API for interacting with the inference endpoint.
• Monitoring and Logging: Monitor the performance and logs of deployed models for effective debugging and optimization.

Getting Started

Prerequisites

• Python >= 3.7, < 3.11
• Docker (if using containerized deployment)

Installation

You can install FlockServe from PyPI with pip:

pip install flockserve

Usage

Running from command line:

flockserve --skypilot_task serving_tgi_cpu_openai.yaml

From Python:

from flockserve import FlockServe
fs = FlockServe(skypilot_task="serving_tgi_cpu_generate.yaml")
fs.run()

The mandatory argument is skypilot_task. The available arguments are:

Once FlockServe is started, it will print the outputs from SkyPilot, as well as report FlockServe metrics periodically:

INFO:flockserve.flockserve:Workers: 1, Workers Ready: 0, Worker Load: 0, QLRM: 0.0

Once "Workers Ready" is more than 0, you can send requests:

curl -X POST -H "Content-Type: application/json" 0.0.0.0:3000/v1/chat/completions -d "@server_test_tgi_openai.json"

Acknowledgments

FlockServe was developed at JDoodle, the AI-powered online platform for coding.