narwhal-inference 0.1.1

Documentation | Deploy | Architectures | Configuration | API | KPIs | Observability | Benchmarking

Narwhal is a serving framework for disaggregated LLM inference. It re-splits a fleet's prefill/decode roles in seconds, and the weights stay resident. The Price of Order in Disaggregated Inference (Georgiou, 2026, in preparation) is its evidence base. Its scheduling core implements and extends the algorithms from the Arrow paper (arXiv:2505.11916).

About

In most disaggregated fleets, roles are wiring. Prefill nodes sit here, decode nodes sit there, and changing the split means draining hardware. In Narwhal, roles are labels. Every engine can serve either phase. A controller rewrites the labels when the workload shifts, and a re-split settles in seconds with the weights resident.

The scheduling core is an independent implementation of the Arrow paper (arXiv:2505.11916): Algorithms 1-3, the lexicographic cost pairs, event-fed monitoring, stateless instances. Around that core, Narwhal is its own system:

Use Narwhal when the phase mix of your workload moves and a pinned split loses goodput. Your engines must meet the contract the Arrow paper asks for: stateless instances with any-peer KV transfer.

Quick Start

Option A: the demo (CPU only). The simulator replays a 90-second moving trace across the topology spectrum and prints the comparison table in under a minute. It needs only Python 3.11+:

git clone https://github.com/athrael-soju/Narwhal
cd Narwhal
make demo

Option B: a fleet with no GPU. make setup installs the router. make stub-fleet starts six processes that speak the engine protocol on the Arrow paper's timing model, and the router runs against them end to end: profile, gates, serving, and KV handoff.

Option C: real engines. Deploy goes from install to serving against stateless vLLM engines with NIXL kv_both in six steps.

Installing puts these commands on the path:

make check runs the whole suite. Each test names the Arrow paper clause it holds the code to.

Evidence

Narwhal ships the instruments, and the study ships the numbers. The bench drives load and journals every request, the report tool scores goodput, adaptation lag and thrash from the journal alone, and the gates make a run's preconditions explicit. The measured comparison of serving architectures and controllers is the subject of The Price of Order in Disaggregated Inference, and the study's artifact carries the full reproduction chain, from the methodology and the experiments ledger to the raw journals and the campaign drivers. A preprint link will land here on publication. To measure your own fleet, start at Benchmarking. make demo replays a 90-second trace on CPU and shows the shape of the claim in under a minute.

Documentation

docs/ is the operator documentation and the wiki. make wiki mirrors it, so edits belong in docs/, where they ride the same review as code.

• Deploy - install to serving, plus the engine compatibility notes; your own hardware and model start at presets/README (copy presets/_template/, then profile, check, calibrate, serve)
• Configuration - every field, with its validation
• API - every route, and the journal contract
• Serving KPIs - TTFT, TPOT, goodput, and operational diagnostics
• Observability - every metric, alert rule, and board panel
• Benchmarking - measure your own fleet and score the journal
• Architectures - the four fleet designs and the case for hot-swap

Status

Narwhal is an independent implementation and is not affiliated with the Arrow authors. The Arrow paper is the specification for the scheduling core. The router's only hardware contract is the engine's: stateless vLLM instances with NIXL kv_both. The router authenticates nothing by default (tenant keys are the optional door) and assumes a trusted fabric. This is 0.x software, so interfaces may move.

Contributing

CONTRIBUTING.md lists the invariants a change must hold. SECURITY.md states the disclosure policy. Bugs and questions go to issues.

Citation

Cite the software for Narwhal itself, the study for the evidence once it publishes, and the Arrow paper when referencing the algorithms the scheduling core implements. GitHub's "Cite this repository" button reads CITATION.cff.

@software{narwhal,
  author  = {Georgiou, Athos},
  title   = {Narwhal: adaptive hot-swap disaggregation for LLM inference},
  year    = {2026},
  version = {0.1.1},
  url     = {https://github.com/athrael-soju/Narwhal},
}

@unpublished{georgiou2026priceoforder,
  author = {Georgiou, Athos},
  title  = {The Price of Order in Disaggregated Inference},
  year   = {2026},
  note   = {In preparation},
}