narwhal-inference 0.2.1

Documentation | Deployment | API reference | Issues | Contributing

About

Narwhal is a disaggregated LLM inference framework, which reallocates prefill and decode roles as demand changes while model weights stay loaded.

Narwhal provides:

• Hot-swap prefill/decode role assignment across a fixed GPU fleet.
• Separate prefill and decode routing with NIXL KV transfer.
• Latency-aware admission and placement using measured per-engine profiles.
• Streaming and non-streaming completion and chat APIs, including function tools and reasoning output where supported by the engine and model.
• Request deadlines, disconnect cancellation, bounded queues and optional retries.
• Engine health checks, transfer validation and warm-standby router failover.
• Prometheus metrics, request journals and a Grafana dashboard.

Architecture

On regular controller passes, Narwhal prices the current and adjacent prefill/decode splits from measured engine curves, offered demand, and resident work, moving an eligible engine when a candidate improves the worst projected SLO ratio by the configured margin and passes role-floor, cooldown, and health checks. New requests follow the revised split while resident requests finish on their assigned engines.

See Core concepts for request flow and scheduling, and Configuration for controller settings.

Benchmark snapshot

AlPerf v0.12.0 ran chat/document and mixed-payload Kimi-K3 workloads with prefix caching enabled across Narwhal, Dynamo Planner, and Ray Serve LLM.

Install from PyPI

Install the router commands on Linux with Python 3.11 or newer:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install narwhal-inference
narwhal-check --help

The PyPI installation guide covers version checks and the fleet inputs needed after installation. A production deployment also uses an approved source checkout for host preparation and engine launch.

Deploy a fleet

From a management workstation, Deploy a fleet uses the private .env and host inspection to prepare router and GPU engine hosts, launch vLLM with NIXL, and verify a completion through Narwhal. Measure the workload through the private SSH route, reconcile the results, and inspect the fleet through Prometheus and Grafana.

Documentation

• Architecture and scheduling
• Fleet configuration
• HTTP API
• Fleet measurement
• Ingress, monitoring and maintenance
• Troubleshooting

Contributing

Contributing covers checkout setup, local checks and the pull request flow. Participation follows the code of conduct, and the security policy covers vulnerability reports.

Attribution and citation

Narwhal's scheduling algorithms derive from Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture by Wu et al. (2025). Cite Arrow for those algorithms and Narwhal for this software. CITATION.cff contains both references.

License: Apache-2.0.