moe-l2

Run large MoE models on consumer GPUs. A transparent proxy that predicts which experts your prompt needs, preloads them into a shared-memory LRU cache, so you can run 16 GB+ models on 8 GB GPUs.

How it works

MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.

user → moe-l2 proxy (localhost:11435)
    ├── predict domain
    ├── preload domain experts → /dev/shm/moe_l2/
    └── forward to ollama (localhost:11434)

Quick start

pip install moe-l2                   # keyword-only predictor (zero extra deps)
pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
moe-l2 start --model model.gguf --l2-size 4GB

Usage

1. L2 proxy (recommended)

Start the transparent proxy — sits between your client and ollama:

moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --l2-size 4GB

All default ollama tools work through it (curl, open-webui, langchain):

# streaming
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"write a Python script"}],
  "stream":true
}'

# blocking
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"hello"}],
  "stream":false
}'

2. Monitor cache stats

moe-l2 stats --port 11435

Example output:

moe-l2 cache stats
  requests:     47
  hits:         42     (89.4%)
  misses:        5
  slots_used:  32/48  (66.7%)
  memory:     456 MB  (68.3% of 668 MB)

3. Use as a library

from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
from moe_l2.cache import L2Cache

# Predict domain (zero-dependency mode)
domain = predict("print hello world")  # → "codegen"

# Hybrid mode (falls back to semantic embedding)
from moe_l2 import enable_semantic
enable_semantic()
domain = predict_hybrid("deploy nginx on ubuntu")  # → "chinese_tech"

# Setup L2 cache
cache = L2Cache(
    model_path="/models/model.gguf",
    slots_per_layer=48  # auto-calculated from --l2-size
)
expert_map = load_mapping()
cache.preload_domain("codegen", expert_map)

# Stats
cache.stats()  # → {"hits": ..., "misses": ..., ...}

Supported domains

Architecture

┌────────────────────────────────────────────────┐
│         Ollama Client (user-facing)             │
│    curl / open-webui / langchain / any tool     │
└──────────┬─────────────────────────────────────┘
           │ POST to :11435
┌──────────▼─────────────────────────────────────┐
│            moe-l2 proxy                         │
│  ┌─────────────────────────────────────────┐    │
│  │ Domain Predictor                        │    │
│  │  ┌──────────┐   ┌───────────────────┐   │    │
│  │  │ Keywords │ → │ Semantic (opt)    │   │    │
│  │  │ ~210词    │   │ all-MiniLM-L6-v2 │   │    │
│  │  └──────────┘   └───────────────────┘   │    │
│  │         ↓ domain                          │    │
│  │  ┌──────────────────────────────────┐    │    │
│  │  │ L2 Cache (LRU + mmap /dev/shm/) │    │    │
│  │  │  preload domain experts async   │    │    │
│  │  └──────────────────────────────────┘    │    │
│  └─────────────────────────────────────────┘    │
│           │ POST to :11434 (transparent)        │
└──────────┬─────────────────────────────────────┘
           │
┌──────────▼─────────────────────────────────────┐
│           Ollama / llama.cpp                    │
│    MoE inference with hot-cached experts        │
└────────────────────────────────────────────────┘

CLI reference

Options:

• --model auto: scan /opt/data/models/*.gguf
• --l2-size 4GB / --l2-size 512MB: target cache size
• --port 11435 (default)
• --gpu: enable GPU mode (requires CUDA + NVIDIA GPU)

GPU binaries: The repo does not track 500MB+ .so files. When you pip install moe-l2, binaries are included. For git-clone users, run moe-l2 download-bins to fetch them from GitHub Release.

Project status

Phase 2 — core components complete (2026-07-30):

• ✅ Domain predictor (keyword + optional semantic)
• ✅ L2 cache (mmap LRU, thread-safe, async preload)
• ✅ GGUF weight reader (direct memmap from .gguf)
• ✅ Transparent proxy (HTTP/SSE forwarding, predict+preload)
• ✅ CLI (start/stats with auto model detection, --gpu support)
• ✅ GPU end-to-end pipeline verified (DS-V2-Lite on RTX 4090 24GB, ~1.6 GiB VRAM)
• ✅ Bundled GPU binaries (A3-patched llama-server + CUDA libs, 532 MB)
• ✅ PyPI v0.2.0 → v0.3.0 release
• 🔲 llama.cpp C++ integration (direct mmap from L2 cache) — Phase 3
• 🔲 GPU LRU expert cache (keep hot experts in VRAM, reduce PCIe transfers)

License

All Rights Reserved. This software is proprietary and confidential. No part may be reproduced, distributed, or transmitted without prior written permission. Copyright (c) 2026 yalun753.