Aegaeon

Serve many LLMs on a shared GPU pool with token-level autoscaling.

Paper · Documentation

Why Aegaeon?

Model-as-a-Service platforms must serve a long tail of LLMs with sparse and unpredictable traffic. Existing GPU pooling approaches either colocate models, which GPU memory limits to two or three models per GPU, or autoscale at request boundaries, where many concurrently active models cause head-of-line blocking.

The SOSP '25 paper Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market presents Aegaeon, a multi-LLM serving system that moves model autoscaling to token granularity. Its token-level scheduler and full-stack optimizations for preemptive model switching enable up to seven models per GPU. In evaluation, Aegaeon sustained 2–2.5× higher request arrival rates or 1.5–9× higher goodput than existing systems; in its Alibaba Cloud Model Studio beta deployment, it reduced the GPU count from 1,192 to 213, an 82% saving.

Key Features

• Token-level autoscaling — switches among active models between token steps instead of waiting for whole requests to finish.
• SLO-aware scheduling — coordinates Prefill and Decode work against TTFT and TPOT objectives.
• Fast model switching — combines host-side weight caching, prefetching, a Rust quick loader, and explicit KV-cache movement.
• Multi-model runtime — supports dynamic model deployment, multiple replicas, and load-aware request routing.
• vLLM backend — reuses model implementations and compute kernels across dense, hybrid, and MoE architectures.
• Serving and observability — provides OpenAI-style Chat and Completion APIs, SSE streaming, and runtime state.

Demo

The demo compares Aegaeon with a request-level baseline while both serve the same two models. It shows model switches and streamed outputs.

Quick Start

Install the CUDA 12.9 release from PyPI

In a fresh Python 3.10 environment on Linux x86_64 with glibc >=2.31, install the published version and its matching CUDA 12.9 dependencies with one command:

python -m pip install aegaeon==0.1.0 --extra-index-url https://download.pytorch.org/whl/cu129 --extra-index-url https://wheels.vllm.ai/568afb3a13806beb53bb2e6bd518269357b237c0/cu129

The extra indexes are the official PyTorch CUDA 12.9 index and the pinned vLLM CUDA 12.9 build index. Aegaeon is downloaded from PyPI; pip installs all declared runtime dependencies automatically. No repository checkout or requirements file is needed for this installation. Keep both indexes: the CUDA variants pinned by this release are not all available on the default PyPI index.

The wheel contains Aegaeon's precompiled CUDA and Rust extensions. Installing it does not require Aegaeon's CUDA/C++/Rust build toolchain. A compatible NVIDIA driver is required. This release was built for compute capability 8.0 and validated on NVIDIA A100; other GPU architectures and the CUDA 13.0 ARM wheel are not validated. See the installation guide for source builds and checks.

Build from source

Use Linux x86_64 and CPython 3.10. Prepare the matching CUDA Toolkit, C++ compiler, and Rust/Cargo as described in the installation guide, then run from the repository root in a fresh environment.

Choose one CUDA version:

# CUDA 12.9
python -m pip install -r requirements-cu129.txt
AEGAEON_CUDA=cu129 python -m pip install --no-build-isolation -e .

# CUDA 13.0 (GPU serving still requires validation)
python -m pip install -r requirements-cu130.txt
AEGAEON_CUDA=cu130 python -m pip install --no-build-isolation -e .

Follow the Quickstart to prepare a model and send your first request. Driver requirements and manual toolchain commands are in the installation guide.

Documentation