StreamLLM

StreamLLM optimizes inference memory usage, allowing large language models (such as 14B, 32B, and 70B models) to run on consumer GPUs with as little as 4GB or 8GB VRAM without requiring distributed hardware.

Instead of loading the entire model into GPU memory at once, StreamLLM dynamically streams transformer layers between system RAM/disk and GPU VRAM, using asynchronous prefetching and double buffering so computation overlaps with data movement.

Quickstart

1. Install package

pip install streamllm

2. Discover and install a model

# View available curated models and installation status
streamllm list

# Download and install a model to your local cache
streamllm install smollm-135m
# Or install mid-sized/larger models:
streamllm install qwen-0.5b
streamllm install qwen-14b

3. Inference via Python SDK

Run inference with automatic layer streaming:

from streamllm import AutoModel

# Load any installed preset or HuggingFace repo ID
model = AutoModel.from_pretrained("smollm-135m")

tokens = model.tokenizer("Hello, what is layer streaming?", return_tensors="pt")
output = model.generate(tokens["input_ids"], max_new_tokens=40)

print(model.tokenizer.decode(output.sequences[0]))

You can pass in any preset alias, HuggingFace model repo ID, or local directory path.

Model Management CLI

StreamLLM provides full model management (like Ollama or Docker) for discovering and downloading open-source weights:

# List all available presets, parameters, memory specs, and install status
streamllm list

# List only locally installed models
streamllm list --installed

# Download & install a model (preset alias or HuggingFace repo ID)
streamllm install qwen-0.5b
streamllm install HuggingFaceTB/SmolLM2-135M-Instruct
streamllm pull llama3.2-1b

# Remove an installed model from local cache
streamllm models remove qwen-0.5b

Curated Model Presets

Interactive Chat & One-Shot Run

StreamLLM includes an interactive terminal chat REPL and command-line runner with real-time token streaming:

# 1. Interactive Chat REPL
streamllm chat smollm-135m
streamllm chat qwen-0.5b
streamllm chat qwen-14b

# If you don't provide a model name, StreamLLM automatically selects your installed model!
streamllm chat

# Inside chat, you have built-in slash commands:
# /stats   - Check live GPU VRAM & scratchpad allocation
# /system  - Update the system instruction prompt
# /clear   - Clear conversation history
# /exit    - Quit chat

# 2. One-shot command execution
streamllm run smollm-135m "Explain layer streaming in 2 sentences"
streamllm run qwen-0.5b "Summarize quantum computing" --max-tokens 100

# 3. Pipeable raw output for scripts
streamllm run smollm-135m "Generate JSON" --raw

# 4. Check GPU VRAM and live PCIe bandwidth
streamllm hardware

# 5. Run streaming micro-benchmark
streamllm bench --layers 16 --hidden-dim 2048 --seq-len 128

How It Works

During inference, StreamLLM keeps only the currently executing layer on the GPU while asynchronously prefetching upcoming layers over PCIe DMA:

• Static GPU Scratchpad Pool (GPUScratchpadPool): Pre-allocates two static VRAM buffer slots (Slot A and Slot B). Eliminates cudaMalloc and cudaFree churn during generation.
• Dual CUDA Streams: Dedicated compute_stream and transfer_stream with lock-free hardware event synchronization (torch.cuda.Event).
• Pinned Host Memory: Uses page-locked RAM (torch.Tensor.pin_memory()) for true non-blocking PCIe transfers.
• KV-Cache Manager: Deterministic memory budgeting that ensures context length growth never causes an Out-of-Memory (OOM) crash.

Benchmark Performance

Measured Hardware: NVIDIA GeForce RTX 3050 Laptop GPU (VRAM: 4096 MB, PCIe: 8.65 GB/s)

Key Takeaway: Even as total model weights scale past 2.0 GB, StreamLLM's static scratchpad pool holds physical GPU memory strictly at 128.0 MB while asynchronous prefetching delivers up to 1.28x faster inference.