WarmSwap

Move a running LLM generation from one GPU to another without losing it.

When you drain a GPU worker, every in-flight session dies with it. The KV cache — everything the model has read so far — is gone, and the user waits through a prompt they already paid for. WarmSwap moves the cache instead: the session continues on the next GPU at the next token, and the client streaming it sees no gap.

Measured on 2× RTX 3090 with vLLM 0.11.0 and Qwen2.5-7B-Instruct, a resumed session is bitwise identical to one that never moved — zero differing KV positions, maximum logit delta 0.0 across 80 concurrent probes.

→ warmswap.github.io

This package

This release reserves the name. It installs no product code.

WarmSwap is source-available and distributed under licence, not from this index: production use requires a written licence and a certificate for the specific deployment being run, because a migration is only exact when both GPUs run the same execution plan. See the licence and the quickstart.

Installing it is not how you get WarmSwap. Read the quickstart and get in touch.