narwhal-inference 0.7.0

What is Narwhal?

Narwhal is an adaptive, disaggregated inference framework that automatically hot-swaps prefill and decode roles as demand changes, and without having to reload model weights. It can scale from a single GPU to multi-node deployments.

Capability	Behavior	Guide
Role hot-swap	Reassigns prefill and decode roles across a fixed GPU fleet, with NIXL key-value (KV) transfer between them
Serving	Serves completion and chat requests, streamed or buffered, with latency-aware admission
Fault tolerance	Fails over to a warm-standby router and readmits engines against their live process generation
Measurement	Profiles engines, runs ordered benchmark points, and keeps the evidence from each point
Observability	Exports router and engine metrics to Prometheus and a provisioned Grafana dashboard
Operator tooling	Validates fleet files offline and collects private diagnostic bundles
Development mode	Runs two to eight engines on one NVIDIA CUDA GPU under Ubuntu or WSL2

How engines change roles

The role controller scores the current role split and each adjacent split, one engine move away. It uses measured engine profiles, offered demand, and resident work, and moves one engine only when the worst projected SLO ratio improves and every role, lifecycle, and resident-work guard passes. New requests follow the revised split; resident requests finish on their assigned engines.

Evaluation

Read the full evaluation: Evaluating Narwhal

Getting the commands

Install on Linux with Python 3.11 or newer:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install narwhal-inference
narwhal-serve --version
narwhal --help

The wheel installs these commands:

narwhal, narwhal-engine, narwhal-serve, narwhal-attest, narwhal-profile, and narwhal-check. See Installing from PyPI and the CLI reference for their options.

Trying it on one GPU

Narwhal dev runs a local NVIDIA CUDA fleet on Ubuntu, either directly or under WSL2.

narwhal dev init
narwhal dev up
narwhal dev verify
narwhal dev status
narwhal dev down

The default template starts two vLLM engines and the SGLang template starts two SGLang engines, each on an NVIDIA GPU whose free VRAM is at least 80% of its total plus 512 MiB. The RTX 5090 reference template starts four vLLM engines on an RTX 5090.

Bringing up a fleet

Run the deployment gates in Deploying a fleet from a management workstation.

Documentation

The MkDocs site routes installation, deployment, operation, measurement, and reference tasks.

Contributing

Built on Arrow

Narwhal's scheduling algorithms derive from Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture by Wu et al. (2025).