narwhal-inference 0.4.1

What is Narwhal?

Narwhal is an adaptive, disaggregated inference framework that automatically hot-swaps prefill and decode roles, without having to reload model weights. It scales from a single GPU to distributed multi-node deployments.

Capability	Behavior	Guide
Role hot-swap	Reassigns prefill and decode roles across a fixed GPU fleet, with NIXL key-value (KV) transfer between them
Serving	Serves streaming and buffered completion and chat requests with latency-aware admission
Fault tolerance	Fails over to a warm-standby router and readmits engines against their live process generation
Measurement	Profiles engines and runs ordered benchmark points with retained evidence
Observability	Exports router and engine metrics to Prometheus and a provisioned Grafana dashboard
Operator tooling	Validates fleet files offline and collects private diagnostic bundles
Development mode	Runs two to eight engines on one NVIDIA CUDA GPU under Ubuntu or WSL2

How engines change roles

The role controller scores the current role split and each adjacent split, one engine move away. It works from measured engine profiles, offered demand, and resident work. The score is the worst projected service-level objective (SLO) ratio across time to first token (TTFT), time per output token (TPOT), and decode queueing.

Demand is the measured window demand, using the larger of the short- and long-horizon decode estimates for decode-to-prefill candidates. The evidence window closes after controller.reactive.evidence_span_s and the minimum arrivals, or after controller.reactive.evidence_max_span_s under sparse traffic.

The controller moves to the adjacent split that improves the score by at least the configured margin. A decode-to-prefill move requires a closed evidence window and stable decode demand, and a prefill-to-decode move proceeds with the window open.

Each move passes the guards for pinned engines, role floors, cooldown, dwell time, the resident-stream ceiling on decode donors, and engine lifecycle holds. Floor repair moves one engine per monitor pass while a phase sits below its configured floor.

New requests follow the revised split, and resident requests finish on their assigned engines.

Evaluation

Read the full evaluation: Evaluating Narwhal

Getting the commands

Install on Linux with Python 3.11 or newer:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install narwhal-inference
narwhal-serve --version
narwhal --help

The wheel installs these commands:

Trying it on one GPU

Narwhal dev runs a local NVIDIA CUDA fleet on Ubuntu or Ubuntu under WSL2.

narwhal dev init
narwhal dev up
narwhal dev verify
narwhal dev status
narwhal dev down

The installed template starts two engines on an NVIDIA GPU with 8 GB of VRAM or less. The RTX 5090 reference template starts four engines on an RTX 5090.

Bringing up a fleet

Run these gates in order from a management workstation:

1. Freeze inputs and discover the deployment in Gate A.
2. Package and install the approved revision in Gate B.
3. Validate and start every engine in Gate C.
4. Qualify the transfer fabric in Gate D.
5. Attest the live engines in Gate E.
6. Profile the engines and run preflight in Gate F.
7. Start the router and validate capacity through an SSH tunnel in Gate G.

Documentation

Contributing

Built on Arrow

Narwhal's scheduling algorithms derive from Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture by Wu et al. (2025).