vLLM IPC Weight Cache (vllm-ipc-cache)

Zero-copy, instant model weight pre-load cache for vLLM on CUDA and Heterogeneous GPUs (such as MetaX MACA).

By decoupling model weight lifecycle from the vLLM engine process, vllm-ipc-cache reduces model restart and loading time from tens of seconds to sub-second (0.47s, 76x speedup) with 0.0 GiB duplicate memory overhead.

🌟 Key Highlights

• ⚡ Sub-Second Weight Attach: Reduces 27B model loading from 35.8s to 0.47s (76x faster).
• 🔌 Zero-Intrusive Plugin Architecture: Packaged as a standard Python wheel using vllm.general_plugins entry points. Works out-of-the-box on older or vendor-forked vLLM without modifying a single line of core vLLM code.
• 🛡️ Enterprise Multi-Tenant Security: Private socket sandbox (0700 dir / 0600 socket) with Linux kernel SO_PEERCRED handshake validation to prevent cross-tenant privilege escalation and handle hijacking.
• 🎯 5-Tuple Fingerprint Guard: The engine handshake validates model path, tensor parallel size & rank, data type, and vLLM version against the daemon's preloaded config. Any mismatch is rejected with a clear error instead of silently attaching wrong weights.
• 🚀 Ideal for Benchmark Sweeps & High Availability: Slashes hours of repeated disk I/O in parameter grid searches (concurrency, batch size, block size sweep) down to minutes.

📊 Benchmark Results

Real-world test on MetaX N300-A GPU (Qwen3.8-27B, Tensor Parallel = 4):

The sweep row is an estimate derived from the measured per-restart saving, not a stopwatch measurement.

🏗️ Architecture

+-------------------------------------------------------------+
|                      GPU Device Memory                      |
|  +-------------------------------------------------------+  |
|  |             Preloaded Weights in VRAM                 |  |
|  +-------------------------------------------------------+  |
|          ^                                     ^            |
|          | (CUDA IPC Export)                   | (Attach)   |
|          |                                     |            |
|   +---------------+                     +----------------+  |
|   | Daemon Worker |                     |  vLLM Engine   |  |
|   +---------------+                     +----------------+  |
|          |                                     |            |
+----------|-------------------------------------|------------+
           |                                     |
           +-----> [Unix Domain Socket] <--------+
                   - 0700/0600 Sandbox
                   - SO_PEERCRED UID Auth
                   - 5-Tuple Fingerprint

1. Daemon Workers load weights into GPU VRAM once and hold them in memory.
2. For each tensor, a CUDA/MACA IPC memory handle is generated.
3. When the vLLM Engine starts with --load-format ipc_cache, it connects to the daemon worker via Unix Domain Socket.
4. After UID authentication and fingerprint validation, the engine maps the IPC handles directly into its memory space via zero-copy.

📦 Installation

pip install vllm-ipc-cache

Note: torch and vllm are expected to be pre-installed in your environment (e.g. vendor container image or standard CUDA env).

🚀 Quick Start

Step 1: Start the Weight Cache Daemon

Launch background daemon workers to preload model weights into GPU VRAM:

# Launch daemon workers (e.g., TP=4 for a 27B model)
python3 -m ipc_cache.daemon \
  --model /path/to/model \
  --tp 4 \
  --master-port 29500

Wait until all TP ranks report READY!. Weights are now resident in VRAM.

Step 2: Start vLLM with IPC Cache

Simply pass --load-format ipc_cache when launching your vLLM engine or OpenAI API server:

python3 -m vllm.entrypoints.openai.api_server \
  --model /path/to/model \
  --tensor-parallel-size 4 \
  --load-format ipc_cache \
  --port 8000

vLLM will automatically detect the plugin and attach to the cached weights in < 0.5s!

🛠️ Typical Use Cases

1. Benchmark Parameter Sweep: Sweep max_num_batched_tokens, max_num_seqs, and block_size without repeatedly waiting for multi-gigabyte disk reads on every restart.
2. High-Availability (HA) Fast Recovery: When an inference worker crashes, the orchestrator restarts the process and restores serving capabilities within a single second.
3. Model Time-Sharing / Dynamic Switching: Swap between active serving engines with minimal cold-start penalty.

🔒 Security & Robustness

• UID Sandboxing: Daemon sockets are created in /tmp/vllm_ipc_cache_uid_<UID> with 0700 permissions. Sockets are restricted to 0600.

• Peer Credential Verification: Before responding to requests, the daemon queries SO_PEERCRED on the incoming socket to verify the connecting PID and UID.

• Fingerprint Verification: Requests must provide a 5-tuple fingerprint matching the daemon's preloaded configuration:

  • Model Path (realpath-resolved)
  • Tensor Parallel Size & Rank
  • Data Type (torch_dtype)
  • vLLM Version

  If any field mismatches, the request is rejected with a clear error message. There is no automatic fallback to cold load — the engine fails loudly instead of silently attaching wrong weights.

⚠️ Limitations & Operational Notes

• Resident VRAM cost: The daemon keeps weights in GPU memory at all times — this design trades VRAM for restart latency. Size your --gpu-memory-utilization with the daemon's share accounted for (e.g. Qwen3.8-27B @ TP=4 keeps ~15 GB per card resident).
• Verified scope:
  • Dense bf16 models (e.g. Qwen3.8-27B): fully verified end-to-end.
  • compressed-tensors W8A8 models (e.g. MiniMax-M2.7, DeepSeek-V4-Flash): architecture & format compatibility verified; full-load benchmarks pending.
• Pipeline parallelism is not supported: the cache is keyed by tensor-parallel rank only (pipeline_parallel_size must be 1).
• SGLang is not supported: the loader hooks vLLM's plugin interface only.
• GPU ordering must match: sockets are keyed by TP rank, so the daemon and the engine must see GPUs in the same order (identical CUDA_VISIBLE_DEVICES mapping).
• Trust model: the socket sandbox and SO_PEERCRED check protect against other local users. Processes running as the same user are within the trust boundary — this matches the upstream vLLM weight-cache threat model.

📄 License

This project is licensed under the Apache-2.0 License.