moe-l2

Run large MoE models on consumer GPUs. A transparent proxy that predicts which experts your prompt needs, preloads them into a shared-memory LRU cache, so you can run 16 GB+ models on 8 GB GPUs.

How it works

MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.

user → moe-l2 proxy (localhost:11435)
    ├── predict domain
    ├── preload domain experts → /dev/shm/moe_l2/
    └── forward to ollama (localhost:11434)

Quick start

pip install moe-l2                   # keyword-only predictor (zero extra deps)
pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
moe-l2 start --model model.gguf --l2-size 4GB

Usage

1. L2 proxy (recommended)

Start the transparent proxy — sits between your client and ollama:

moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --l2-size 4GB

All default ollama tools work through it (curl, open-webui, langchain):

# streaming
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"write a Python script"}],
  "stream":true
}'

# blocking
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"hello"}],
  "stream":false
}'

2. Monitor cache stats

moe-l2 stats --port 11435

Example output:

moe-l2 cache stats
  requests:     47
  hits:         42     (89.4%)
  misses:        5
  slots_used:  32/48  (66.7%)
  memory:     456 MB  (68.3% of 668 MB)

3. Use as a library

from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
from moe_l2.cache import L2Cache

# Predict domain (zero-dependency mode)
domain = predict("print hello world")  # → "codegen"

# Hybrid mode (falls back to semantic embedding)
from moe_l2 import enable_semantic
enable_semantic()
domain = predict_hybrid("deploy nginx on ubuntu")  # → "chinese_tech"

# Setup L2 cache
cache = L2Cache(
    model_path="/models/model.gguf",
    slots_per_layer=48  # auto-calculated from --l2-size
)
expert_map = load_mapping()
cache.preload_domain("codegen", expert_map)

# Stats
cache.stats()  # → {"hits": ..., "misses": ..., ...}

Supported domains

Architecture

┌────────────────────────────────────────────────┐
│         Ollama Client (user-facing)             │
│    curl / open-webui / langchain / any tool     │
└──────────┬─────────────────────────────────────┘
           │ POST to :11435
┌──────────▼─────────────────────────────────────┐
│            moe-l2 proxy                         │
│  ┌─────────────────────────────────────────┐    │
│  │ Domain Predictor                        │    │
│  │  ┌──────────┐   ┌───────────────────┐   │    │
│  │  │ Keywords │ → │ Semantic (opt)    │   │    │
│  │  │ ~210词    │   │ all-MiniLM-L6-v2 │   │    │
│  │  └──────────┘   └───────────────────┘   │    │
│  │         ↓ domain                          │    │
│  │  ┌──────────────────────────────────┐    │    │
│  │  │ L2 Cache (LRU + mmap /dev/shm/) │    │    │
│  │  │  preload domain experts async   │    │    │
│  │  └──────────────────────────────────┘    │    │
│  └─────────────────────────────────────────┘    │
│           │ POST to :11434 (transparent)        │
└──────────┬─────────────────────────────────────┘
           │
┌──────────▼─────────────────────────────────────┐
│           Ollama / llama.cpp                    │
│    MoE inference with hot-cached experts        │
└────────────────────────────────────────────────┘

CLI reference

Options:

• --model auto: scan /opt/data/models/*.gguf
• --l2-size 4GB / --l2-size 512MB: target cache size
• --port 11435 (default)

Project status

Phase 2 — core components complete (2026-07-30):

• ✅ Domain predictor (keyword + optional semantic)
• ✅ L2 cache (mmap LRU, thread-safe, async preload)
• ✅ GGUF weight reader (direct memmap from .gguf)
• ✅ Transparent proxy (HTTP/SSE forwarding, predict+preload)
• ✅ CLI (start/stats with auto model detection)
• ✅ GPU end-to-end pipeline verified (DS-V2-Lite on RTX 4090 24GB, Dec 2026)
• 🔲 llama.cpp C++ integration (direct mmap from L2 cache) — Phase 3
• 🔲 PyPI v0.2.0 release

License

MIT