moe-l2

English | 中文

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. A transparent, OpenAI-compatible proxy that predicts which experts your prompt needs and preloads them into a shared-memory LRU cache, so you can run 16 GB+ MoE models on 8 GB GPUs with up to 91% VRAM savings.

⭐ Found this useful? Give us a star — it helps others discover the project. ★ Star on GitHub

Real-world benchmark

Speed = RTX 4090 measured (2026-08-10, selective pin + A3 cache 2048, multi-arch build); 2080 Ti full-chain: Qwen 52.07 t/s, DS-V2-Lite 94.95 t/s. See models-benchmark.md.

Without moe-l2, an 8 GB card cannot load these models at all — it OOMs immediately. With moe-l2, a 32B MoE fits in ~2.9 GB VRAM (on-demand pin experts on Qwen3.6-A3B, GPU compute). DeepSeek-V4-Flash (157B params / 85 GB file, 256 experts, top-6) runs on a 10-11 GB card at 8.3-9.1 GB VRAM — with selective pin, RSS drops to 10.4 GB (from 84.4 GB, −88%) with zero speed regression (30.9 t/s). Full report: deepseek-v4-flash-verify-20260805.md · All measured models: models-benchmark.md

Benchmarked on RTX 4090 (2026-08-07, on-demand pin main path)

We benchmarked Qwen3.6-A3B (32B MoE) and DeepSeek-V2-Lite (16B MoE, 64 experts) on RTX 4090. 2026-08-07 main path upgraded to on-demand pin (lazy mmap load + first-touch merge-registration of the whole expert tensor + A3 cache 2048 slots): experts stay in CPU RAM (zero VRAM), the scheduler copies only the activated experts to GPU each step, hot experts are cached in VRAM. DS-V2-Lite 12.5 → 37.9 t/s (+200%), Qwen3.6-A3B 10 → 50.2 t/s (+400%, beats pre-lazy 46.5). Full reports: qwen3.6-a3b-iq2m-benchmark.md · deepseek-v2-lite-q2k-benchmark.md · cache-sched-layer-benchmark.md · models-benchmark.md · Why host-buffer? Full approach history: design-decisions_EN.md / design-decisions.md (中文)

Low-memory mode: dynamic pin set (2026-08-09)

Measured on RTX 4090: 105 cross-topic chat rounds, RSS 9.8 → 41 GB capped (peak 45 GB), vs 84 GB whole-pin. Animated: dynpin-curve.gif

Default (no env vars) = whole-pin at full v0.4.0 speed (Qwen 2080 Ti ≈16 t/s, V4 4090 30.9 t/s); set MOE_L2_LRU=1 to enable this low-memory mode. The default on-demand pin registers the whole expert tensor on first touch — fastest but pins ~82 GB of expert pages in RAM for 85 GB models. For memory-constrained machines, dynamic pin set registers only the experts actually activated (per-expert, group-registered), and the LRU evictor unregisters + madvises cold experts. Measured on RTX 4090 / DeepSeek-V4-Flash-UD-IQ2_M (85 GB): RSS 84 GB → 17-24 GB (configurable via MOE_L2_LRU_MAX_EXPERTS), speed 4-5 t/s (new experts pay a first-touch page-fault; V4 routes extremely wide — a 30-turn session touches ~29 GB of distinct experts). Small MoE models (Qwen3.6-A3B / DS-V2-Lite) keep full speed — their working sets fit any budget.

Enable (example, 32 GB machine):

LLAMA_EXPERT_LOG=1 MOE_L2_LRU=1 MOE_L2_LRU_MAX_EXPERTS=12000 \
MOE_L2_EVICT_MB=20000 MOE_L2_EVICT_INTERVAL=4 \
MOE_L2_PIN_LAYERS=0-2,14-20,36-37 GGML_OP_OFFLOAD_MIN_BATCH=1 \
llama-server -m model.gguf -ngl 99 -c 2048 --no-webui

MOE_L2_PIN_LAYERS pins universal/sparse layers forever (L0-L2 + sparse layers = ~5.4 GB free lunch on V4). Tune MOE_L2_LRU_MAX_EXPERTS: 2000 ≈ 17 GB RSS (tight, slower) / 12000 ≈ 24 GB RSS (≈5.3 t/s on V4) / omit to disable eviction. Trade-off summary: whole-pin = fastest (30.9 t/s V4) at 82 GB RAM; selective pin (router-map) = 10.4 GB RSS at 30.9 t/s (new in v0.4.0); dynamic pin = 17-24 GB RAM at 4-5 t/s V4 (small models unaffected).

Multi-architecture binaries (bins-v0.4.0, 2026-08-10)

One binary for all NVIDIA consumer GPUs — GTX 1080 (sm_61) through RTX 50-series (sm_120a). Built with CUDA 12.8; no per-GPU compilation needed. moe-l2 download-bins fetches it automatically. bins-v0.4.0 includes the selective pin (router-map driven) + GPU cache prefill + on-demand pin main path + dynamic pin set (low-memory mode) + expert-page eviction v3.1 (MOE_L2_LRU_MAX_EXPERTS=N) + layered pin (MOE_L2_PIN_LAYERS) + A3 cache 2048 slots + cuda-libs (no libnccl — not needed for single-GPU).

* 4090 measured with the multi-arch package (2026-08-07, on-demand pin + cache 2048, CUDA 12.8); 2080 Ti row = bins-v0.4.0 full-chain (moe-l2 start --gpu, 2026-08-10, +145~730% vs vanilla llama.cpp); 3080 Ti / 5090 rows are v3.1 multi-arch (bins-v0.3.0). Qwen single-turn 24.5 t/s on 2080 Ti (bins-v0.3.2, 2x vs old host-buffer 11.15).

Verified on 2080 Ti (SM75), 3080 Ti (SM86) and 5090 (SM120a) with the multi-arch build. The 3080 Ti run was +55% faster than the previous CUDA 11.8 single-arch build (12.25 vs 7.88 t/s). Note: SM120a (RTX 50) kernel efficiency in llama.cpp 76f46ad is not yet mature — RTX 5090 shows only +36% over 3080 Ti on DS and −27% on Qwen; a newer llama.cpp rebuild should improve 50-series speed. Full report: multi-arch-three-gpu-benchmark.md · DeepSeek V4 Flash (157B) dual-GPU run: deepseek-v4-flash-verify-20260805.md

Quick start

One-line install (Linux x86_64 + NVIDIA GPU):

curl -fsSL https://raw.githubusercontent.com/yalun753/moe-l2/main/scripts/install.sh | bash

The installer checks your GPU/driver/Python, installs moe-l2 from PyPI, downloads the pre-built CUDA binaries, optionally downloads a demo model (Qwen3.6-35B-A3B, ~11.5 GB, resumable), then runs a self-check.

Manual install:

pip install moe-l2                   # keyword-only predictor (zero extra deps)
pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
moe-l2 download-bins                 # pre-built CUDA llama-server (on-demand pin patched)
moe-l2 model download --model qwen3.6-35b   # optional demo model (~11.5 GB)
moe-l2 start --model model.gguf --gpu

Useful commands:

moe-l2 doctor                        # environment self-check (GPU/CUDA/Python/disk)
moe-l2 model list                    # list downloadable models
moe-l2 model download --model <name> # download model (resumable, via hf-mirror)

Your tools (curl, Open WebUI, LangChain) connect to localhost:11435 — no client changes needed.

How it works

MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.

user → moe-l2 proxy (localhost:11435)
    ├── predict domain (keyword → TF-IDF → semantic)
    ├── on-demand pin experts (lazy mmap + register, zero VRAM)
    ├── hot experts cached in VRAM (A3 LRU)
    └── forward to llama-server (localhost:11436, CUDA GPU)
        └── GPU reads pinned experts via PCIe DMA; cold pages evicted

Visual demo (RTX 4090, 2026-08-02)

Qwen3.6-35B-A3B (32B MoE) — standard vs moe-l2	DeepSeek-V2-Lite (16B MoE) — 8 GB card vs 24 GB card

Summary: 93% less VRAM · 58% of full-GPU speed · 3.1× model-per-GB ratio — an 8 GB card runs what used to need 24 GB (measured 2026-08-02 host-buffer build; 2026-08-07 on-demand pin: DS 37.9 t/s @ 2.0 GB / Qwen 50.2 t/s @ 2.9 GB):

Live capture: Qwen3.6-35B-A3B generating 3,200 tokens with VRAM pinned at ~2.4 GB (41.6 t/s) — watch the VRAM curve stay flat below the 8 GB line the whole run:

examples/demo-assets/demo-vram-animation.mp4 (45 s, 1280×720) · raw telemetry: examples/demo-assets/rec_data.csv · full generated text: examples/demo-assets/rec_full.txt

Usage

1. L2 proxy with GPU on-demand pin (recommended)

Start the transparent proxy with the bundled on-demand pin llama-server:

moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --gpu

The proxy exposes OpenAI-compatible endpoints — all your tools work through it (curl, open-webui, langchain):

# streaming
curl http://localhost:11435/v1/chat/completions -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"write a Python script"}],
  "stream":true
}'

# blocking
curl http://localhost:11435/v1/chat/completions -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"hello"}],
  "stream":false
}'

2. Monitor cache stats

moe-l2 stats --port 11435

Example output:

moe-l2 cache stats
  requests:     47
  hits:         42     (89.4%)
  misses:        5
  slots_used:  32/48  (66.7%)
  memory:     456 MB  (68.3% of 668 MB)

3. Use as a library

from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
from moe_l2.cache import L2Cache

# Predict domain (zero-dependency mode)
domain = predict("print hello world")  # → "codegen"

# Or use the hybrid semantic predictor
domain = predict_hybrid("how do I sort a list?")  # → "codegen"

# Preload experts
cache = L2Cache(model_path="model.gguf", l2_size="4GB")
cache.preload(domain_to_expert_ids[domain])

Architecture

┌────────────────────────────────────────────────────────────┐
│                       HTTP client                          │
│     curl / open-webui / langchain / any OpenAI client      │
└──────────┬─────────────────────────────────────────────────┘
           │ POST /api/chat
           ▼
┌────────────────────────────────────────────────────────────┐
│               moe-l2 Proxy (port 11435)                     │
│                                                            │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Domain Predictor                                    │   │
│  │  - Keyword mode: zero deps, instant classification   │   │
│  │  - Hybrid mode: +sentence-transformers for context   │   │
│  └─────────────────────┬───────────────────────────────┘   │
│                        │ domain                             │
│  ┌─────────────────────▼───────────────────────────────┐   │
│  │  L2 Cache (mmap'd shared memory)                    │   │
│  │  - LRU eviction policy                              │   │
│  │  - Async preload: next-prediction prefetch          │   │
│  │  - Thread-safe concurrent access                    │   │
│  │  - Zero-copy mmap from SSD → RAM                    │   │
│  └─────────────────────┬───────────────────────────────┘   │
│                        │ forward request                    │
└────────────────────────┼───────────────────────────────────┘
                         ▼
┌────────────────────────────────────────────────────────────┐
│         llama-server (port 11436, CUDA GPU)                │
│         on-demand pin experts: lazy mmap, zero VRAM       │
│         GPU reads pinned experts via PCIe DMA             │
│         hot experts cached in VRAM (A3 LRU, 2048 slots)   │
│         cold expert pages evicted (v3.1, RSS capped)      │
└────────────────────────────────────────────────────────────┘

CLI reference

Options:

• --model auto: scan /opt/data/models/*.gguf
• --l2-size 4GB / --l2-size 512MB: target cache size (proxy-only mode)
• --port 11435 (default)
• --gpu: enable GPU mode (requires CUDA + NVIDIA GPU; spawns bundled on-demand pin llama-server on 11436)

GPU binaries: Not tracked in git (bundled as llama_bins.tar.gz, ~2.0 GB multi-architecture on the bins-v0.4.0 release — sm_61/75/86/89/120a, one binary for all NVIDIA consumer GPUs, ships cuda-libs). Fetched at runtime via moe-l2 download-bins. When you pip install moe-l2, binaries are included. For git-clone users, run moe-l2 download-bins to fetch them from GitHub Release.

Platform requirements

• Linux x86_64 only — pre-built binaries target Linux AMD64 (CUDA .so + llama-server)
• macOS, Windows, and ARM Linux are not supported
• NVMe SSD strongly recommended
• NVIDIA GPU required for --gpu mode

More data

The speed tradeoff is intentional and small: expert weights live in CPU RAM (lazy mmap, zero VRAM) and are pinned on first touch — the GPU reads them directly via PCIe DMA, hot experts are cached in VRAM (A3 LRU), and cold pages are evicted to keep RSS capped. On the 2026-08-07 on-demand pin build, DS-V2-Lite reaches 37.9 t/s gen at 2.0 GB VRAM — ~58% of full-GPU speed at <9% of the VRAM.

Expert offload & cache fast path (llama.cpp)

Beyond the proxy layer, moe-l2 ships llama.cpp patches that compile expert handling directly into the CUDA backend — no proxy needed. Two mechanisms:

1. On-demand pin expert GPU fast path (recommended, 2026-08-07). Expert tensors live in CPU RAM via lazy mmap (zero VRAM). On first touch during inference the whole expert tensor is merge-registered as pinned (cudaHostRegister), so the GPU reads it directly via PCIe DMA with no per-step copies. Hot experts are cached in VRAM (A3 LRU, 2048 slots) and cold pages are evicted (v3.1) to keep RSS capped. This is what the benchmark above measures (DS 37.9 / Qwen 50.2 t/s at 2.0 / 2.9 GB VRAM).

2. A3 LRU expert cache (historical, --expert-cache). An LRU cache that keeps recent experts on GPU. In the old --cpu-moe CPU-compute architecture it cut VRAM from 6.6 GB → 1.2 GB (5.64×) at 8.2 t/s. In the current on-demand pin architecture the cache is hooked into the scheduler copy layer (GGML_CUDA_EXPERT_CACHE) and only pays off for small, frequently-hit experts (see below).

When the cache helps (and when it doesn't)

The sched-cache only pays off when experts are small and frequently hit. Verified on RTX 4090 (host-buffer, 2026-08-02):

Key findings (2026-08-02, cache hooked into the scheduler input-copy layer):

• The cache sits in copy_experts: on hit it does a D2D copy (no PCIe round-trip), on miss it falls back to the pinned-host CPU→GPU path and writes back. It only intercepts single-expert groups.
• Benefit = expert size × hit rate. DS (1.55 MB, top-6) wins big; Qwen (~1 MB) pays for itself at best; Mixtral (252 MB, top-2) never hits enough to pay for its VRAM slots.
• Recommended: GGML_CUDA_EXPERT_CACHE=0.25 for DS-class models (16 slots/layer cover all hot experts, VRAM unchanged). Leave it off for Qwen/Mixtral.

Run the demo yourself: bash examples/demo_a3_compression.sh (edit paths first).

Related work

AirLLM (lyogavin/airllm, ~29k stars)

AirLLM is a general-purpose layer-offload scheme for very large models. Its scheduling granularity is the full Transformer layer: during inference only one layer's weights stay in VRAM, everything else is swapped to/from disk — giving an extreme low-VRAM floor (4GB GPU runs 70B). But it has three weaknesses: ① every generated token requires reading/writing a full layer to/from disk, so IO cost is huge and interactive speed is very low; ② no MoE-specific routing prediction or expert hot cache (per-expert streaming only started in 2026-07, with Kimi K3), so repeated prompts keep triggering heavy disk reads; ③ built on native Hugging Face Transformers, with no OpenAI-compatible serving interface out of the box, making it awkward to wire into Open WebUI, LangChain, etc.

Which to choose: pick moe-l2 if you run MoE models (DeepSeek/Qwen) locally for chat, have an 8–12GB older NVIDIA card, want an OpenAI API for tooling, or use multi-shard giant GGUFs. Pick AirLLM if you need dense (non-MoE) models, use Windows/macOS/AMD or CPU-only environments (moe-l2 currently requires Linux + NVIDIA), only do one-shot batch generation, or must stay with native HF weights.

Testing

Automated CI runs on every push (GitHub Actions, Python 3.10–3.13): ruff lint, pytest with coverage (fail below 50%), and package build. Status badge:

• 113 tests covering the Python scheduler core: domain predictor (keyword boundaries, fallback), L2 cache (LRU eviction, pinning, domain switching), GGUF weight reader (synthetic models), transparent proxy (live fake-backend HTTP, blocking + SSE), CLI helpers and the training data flywheel.
• Coverage: 72–88% on the core modules (cache 88%, proxy 78%, gguf_reader 73%, predictor 72%), ~55% total.
• Run locally:

  uv sync --group dev
  uv run pytest tests/
  uv run ruff check moe_l2/ tests/
  

The C++ side (llama.cpp on-demand-pin / expert-cache patches) is GPU-bound and is verified by the end-to-end benchmark reports in references/ — see models-benchmark.md.

Project status

• ✅ Domain predictor (keyword + optional semantic)
• ✅ L2 cache (mmap LRU, thread-safe, async preload)
• ✅ Transparent proxy (HTTP/SSE forwarding)
• ✅ CLI with auto model detection, GPU mode, and collect (routing data → expert map)
• ✅ Host-buffer expert GPU fast path (2026-08-02): DS-V2-Lite 12.5 → 37.5 t/s, Qwen3.6-A3B 10 → 46.8 t/s at 1.6 / 2.1 GB VRAM — experts in CPU pinned memory, only activated experts copied to GPU
• ✅ On-demand pin main path (2026-08-07): lazy mmap load + first-touch merge-registration of the whole expert tensor + A3 cache 2048 slots → Qwen 50.2 / DS 37.9 / V4 10.1 t/s on 4090 (V4 5× faster than 1.7-2.0); fixes CUDA 11.8 cross-register-range copy crash
• ✅ Selective pin + GPU prefill (2026-08-10, v0.4.0): router-map-driven top-K pin → V4 RSS 84.4 → 10.4 GB (↓88%) at 30.9 t/s zero regression; GPU cache prefill lifts cold-start round1 10.7 → 19.7 t/s (+84%). Root-cause: the earlier 10.1 t/s V4 number was the vanilla llama.cpp binary — the moe-l2 optimized build was always 30 t/s. 2080 Ti full-chain: Qwen 52.07 / DS 94.95 t/s (+145730% vs vanilla)
• ✅ Expert cache boundary verified on Mixtral 8x7B / RTX 4090 (2026-08-02, sched-cache): cache benefit = expert size × hit rate — DS-V2-Lite (1.55 MB, top-6) gets Prompt +211% / Gen +5% at cache=0.25; Qwen (~1 MB) and Mixtral (252 MB, top-2) get no gain. Recommended: cache=0.25 for DS-class, off otherwise.
• ✅ DeepSeek-V4-Flash (157B MoE) verified (2026-08-05): 85 GB 3-shard GGUF runs on 2080 Ti (11 GB) and RTX 3080 (10 GB) — VRAM 8.3-9.1 GB, RSS capped by expert-page eviction v3.1 (fixed-expert-count LRU, MOE_L2_LRU_MAX_EXPERTS), multi-shard GGUF parsing fix shipped. Speed 0.89-2.22 t/s (GPU compute bound). Full report
• ✅ PyPI package (moe-l2)

License

Apache 2.0. See LICENSE for details.