moe-policylang 1.12.0

A domain-specific language for Mixture-of-Experts scheduling policies

MoE-PolicyLang

A scheduling language for Mixture-of-Experts models.

Author: Jesse Pokora · License: MIT

---

Install and run — no configuration required

pip install moe-policylang[gpu]

import moe_policylang
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0924")
tok = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0924")

# auto_attach inspects the model + GPU and generates a tuned policy
# automatically.  Capacity, eviction, prefetch budget — all derived
# from num_experts, top_k, expert size, and available VRAM.
mgr = moe_policylang.auto_attach(model)

output = model.generate(**tok("Hello", return_tensors="pt").to(model.device))
print(mgr.get_stats())  # hit rate, transfers, evictions

That's the whole minimum. No .policy file, no cache block, no prefetch block — auto_attach picks sensible defaults from your specific model and GPU. When you want to write a policy explicitly, you can; see The Language below.

---

TL;DR

MoE models pack a lot of weights into "experts" but only fire a few per token. The rest can live in CPU RAM and get pulled to the GPU on demand. Which experts to keep, when to prefetch them, and what to do on a miss is a policy question, and every existing MoE serving system hardcodes its answer inside the runtime.

MoE-PolicyLang is a small declarative language for that policy. Swapping LRU for LFU is a one-word change; nothing else has to move.

On an RTX 5080 (16 GB), it runs Qwen1.5-MoE (28.6 GB fp16) at 4.61 tok/s — 8.1x faster than HuggingFace's device_map="auto" — with bit-identical output.

For researchers (v1.5.3+): the same package ships a routing diagnostics + offline replay surface — static routing_profile() to predict whether a model's softmax is sharp enough for score-based prefetching, validate_policy(ir, accessor=...) warnings for architecture/policy mismatches, and record_trace + replay_policies / replay_structures for sweeping new cache and prefetch ideas against a recorded trace without re-running the model. See docs/RESEARCH.md.

---

Manuals

• docs/MANUAL.md — Language Manual. Full reference for the .policy DSL and Python eDSL: policy axes (cache / prefetch / schedule / monitor / per-layer / adapt), validation rules, HuggingFace + vLLM integration, autotuner, and API reference.
• docs/RESEARCH.md — Research Workflow Guide (v1.5.3+). Routing diagnostics (static + empirical), the architecture-compatibility validator, and the offline trace recorder/replay benchmark for comparing new cache and prefetch policies against the eight built-ins (including a Belady oracle) without re-running the model.

---

The problem

A Mixture-of-Experts layer replaces a single feed-forward block with N "experts" plus a small router. For each token the router picks the top-k experts (typically k=2 to k=8) and only those fire. Mixtral uses 8 experts per layer (top-2), Qwen1.5-MoE uses 60 (top-4), DeepSeek-V2-Lite uses 64 (top-6). Most experts sit idle on any given token, but the weights still have to be reachable in case the router picks them.

For a 28 GB MoE model on a 16 GB GPU, that means offloading: keep some expert weights on GPU, the rest in CPU RAM, and move weights across the PCIe bus when the router asks for one that isn't resident. PCIe is much slower than reading from GPU memory, so if every router pick is a miss you're stuck — HuggingFace's device_map="auto" runs Qwen1.5-MoE on a 5080 at 0.57 tok/s.

There are four interlocking decisions to make:

Decision	Question	Example strategies
Cache	Which experts stay on GPU?	LRU (drop least-recently-used), LFU (drop least-frequently-used), score-based
Prefetch	Which to load before they're requested?	Affinity (layer L → L+1 patterns), history, lookahead
Schedule	What to do on a cache miss?	Wait for the GPU transfer, run on CPU, decide per-call
Adapt	When to change strategy?	Conditional rules on runtime metrics

Every existing MoE serving system (ExpertFlow, Fiddler, MoE-Infinity, HybriMoE, ProMoE, FineMoE) hardcodes these four decisions inside its runtime. Changing any one strategy means reading and modifying the system's expert-management module: roughly 200 to 2,000 LOC depending on the system.

MoE-PolicyLang lets you write the policy as a short .policy file and attach it to a HuggingFace or vLLM model. The runtime hooks that consume the policy stay the same; only the policy text changes.

---

The Language

A MoE-PolicyLang policy is a .policy file with four composable blocks:

policy balanced {
    cache {
        capacity = 16
        eviction = lfu
        frequency_decay = 0.9
    }
    prefetch {
        strategy = history
        budget = 4
    }
    schedule { mode = hybrid }
    adapt {
        when hit_rate < 0.4 for 100 accesses
            { eviction = lru }
    }
}

Block	Controls	Strategies
cache	Which experts stay on GPU	LRU drops the least-recently-used; LFU drops the least-frequently-used with decay; score ranks by router gate value; freq-threshold keeps anything above a frequency cutoff
prefetch	Experts loaded before they're requested	History uses a running co-occurrence matrix; affinity uses layer L → L+1 patterns; lookahead peeks ahead in the router output
schedule	What happens on a cache miss	gpu-only waits for the transfer; cpu-fallback runs the missed expert on CPU; hybrid decides per-call based on estimated latency
adapt	Self-tuning at runtime	Conditional rules of the form when <metric> <op> <value> for <window> { <override> }

Switching from LRU to LFU is a one-word change. Adding prefetching is two lines.

---

Attaching a policy explicitly

The quick-start at the top uses auto_attach, which generates a tuned policy from the model + GPU. When you want full control over the policy, attach a .policy string directly:

mgr = moe_policylang.attach(model, """
    policy aggressive {
        cache { capacity = 8  eviction = lru }
    }
""")

Or load one from a file:

mgr = moe_policylang.attach(model, open("my_policy.policy").read())

auto_attach is itself a thin wrapper that picks one of four named strategies (aggressive / balanced / conservative / hw_limit) and hands the generated DSL to attach(). To preview what it would generate without attaching, call auto_policies(model) and inspect the strings.

attach() parses the policy, runs the validator, compiles it to a PolicyHook, and registers forward hooks on every MoE layer of the model. From that point on, normal model.generate() calls trigger the hooks.

---

How a policy runs at runtime

On each MoE layer, after the router picks its top-k experts:

1. Look up each picked expert in the GPU cache (hit or miss).
2. For misses, decide whether to wait for a CPU→GPU transfer or run that expert on CPU. The schedule block picks the policy.
3. Insert newly loaded experts into the cache. If full, the eviction rule drops something.
4. Prefetch experts the next few layers are likely to want.
5. Run memory-pressure and TTL eviction triggers.

The hook is plain Python. It adds 6 µs/layer (simple LRU) to 47 µs/layer (composed policy with triggers), against an MoE forward-pass baseline of about 1,500 µs on A100 — under 3.2% of layer time.

The five steps above are the always-on layer of dynamism: cache contents change on every dispatch. PLCB adds a second, opt-in layer — set rebalance_interval > 0 in a per_layer block and the per-layer budget itself drifts at runtime based on observed routing entropy. adapt blocks add a third, opt-in layer — a threshold like when hit_rate < 0.4 rewrites the policy in place (deep-copy IR, validate, recompile, hot-swap the hook) without pausing inference. You pick how much dynamism you want; uniform allocation with no adapt rules is the static default.

---

Why a Language, Not YAML?

The cache, prefetch, and schedule blocks are key-value config and you could handle them with a JSON schema. The reason the grammar pays off is PLCB and the adapt block.

PLCB describes a per-layer cache layout, not a single cache:

per_layer {
    allocation         = uniform
    total_budget       = 864
    rebalance_interval = 500
    min_capacity       = 4
    max_capacity       = 48
}

"27 separate caches at total budget 864 with optional entropy allocation" is awkward to write as flat key-value pairs. The DSL gives it a block.

adapt is the other one. Hot-swap rules monitor metrics and rewrite the policy at runtime:

adapt {
    when hit_rate < 0.4 for 100 accesses { eviction = lru }
}

That's a conditional, not a config value. The grammar constrains what you can write (no arbitrary code in a scheduling policy), and 20 semantic rules catch bad policies at parse time instead of mid-inference.

A Python eDSL (@sched.policy decorator) and an auto_attach API are also available for programmatic construction and zero-config deployment.

---

Results

Live inference on consumer GPU

When the model doesn't fit: Qwen1.5-MoE-A2.7B (~28.6 GB fp16) on RTX 5080 (16 GB VRAM). Without MoE-PolicyLang, the only option is device_map="auto" at 0.57 tok/s. With a 4-line DSL policy:

Config	Strategy	Cap	VRAM	tok/s	95% CI
Baseline (auto)	—	—	12.0 GB	0.57±0.00	—
Skeleton	LRU (cap=1)	1	4.7 GB	4.23±0.22	[4.04, 4.42]
Aggressive	LRU	2	5.2 GB	4.17±0.03	[4.14, 4.20]
Balanced	LFU+hist.	4	7.3 GB	4.35±0.06	[4.30, 4.40]
Generous	LFU+hist.	8	10.1 GB	4.61±0.08	[4.54, 4.68]

Cost-performance: 4.61 tok/s on a $1k consumer GPU is comparable in absolute throughput to Fiddler's 4.17 tok/s on a $15k A100, though the models are different and the numbers are not directly comparable.

Decomposition: roughly 92% of the 8.1x speedup comes from expert-aware loading (skeleton on GPU, experts on CPU). Even a capacity-1 "every dispatch is a miss" config reaches 4.23 tok/s (7.4x). Caching adds the remaining +0.38 tok/s. The DSL's contribution is not the loading mechanism (any system could implement that) but the remaining 8%: the policy layer that decides what to cache, evict, and prefetch, accessible without runtime modification and adaptable at runtime via adapt rules that no static config expresses. On models with more experts, the policy layer's share grows. On DeepSeek (A100), matched-budget per-layer allocation gains +14.7pp hit rate over flat caching at the same total slot count, a pure policy-structure effect with no capacity confound.

n=5, bootstrap 95% CIs. For output correctness: greedy decoding (do_sample=False) produces bit-identical token sequences across all policy configs vs device_map="auto" baseline (4 prompts x 3 policies = 12 comparisons); perplexity on wikitext-2 matches within 0.024%.

When the model fits (overhead measurement): OLMoE-1B-7B (~14 GB) fits entirely on 16 GB VRAM. There vanilla (no hooks) is fastest at 39.2 tok/s, with the policy hooks adding 12-14% overhead. This is not the target scenario; it measures overhead when there is nothing to offload. MoE-PolicyLang is for models that don't fit.

---

PLCB: Per-Layer Cache Budgeting (with a negative result)

A flat cache holds, say, 32 expert weights total, shared across all 27 MoE layers of DeepSeek-V2-Lite. If layer 0 is hot, LFU keeps its experts, and layers that haven't appeared recently get evicted. In steady state, a flat cap=32 cache on DeepSeek covers only ~11 of the 27 layers — the other 16 have zero experts cached and miss every dispatch.

A per-layer cache splits the same total budget (864 = 27×32 slots) into one cache per layer. Each layer keeps its own hot experts.

There's a positive result and a caveat.

The caveat first: per-layer caching only helps when each layer's budget covers that layer's working set. On 16 GB consumer hardware the per-layer budgets are too small for that, and the aggregated cache pushes the CUDA allocator to the VRAM ceiling — throughput drops by about 16%. Flat shared caching is the default for memory-constrained deployments. Per-layer wins with VRAM headroom and high expert counts (DeepSeek-V2-Lite on A100, below).

2. When the regime permits, per-layer cache structure is what matters. At matched total budget on DeepSeek-V2-Lite (A100-80GB), replacing a shared cache with per-layer caches gives +14.7pp hit rate in offline trace replay and eliminates all CPU/GPU transfers in steady state. Output is bit-identical to the fully-resident baseline.

The headline throughput gain (1.60 to 10.22 tok/s, +540%) compares shared-32 to per-layer-864, which is 27x more total slots. The matched-budget +14.7pp hit rate and transfer elimination are the core findings; the 540% wall-clock number folds in the capacity expansion.

This structural difference maps directly onto MoE-aware baselines. Fiddler's expert placement is a hardcoded global popularity ranking, which is structurally equivalent to the flat cache on the left. On an A100-80GB where 85% of Mixtral's experts fit on-device (217/256), the ranking barely matters because almost everything is resident. On a constrained GPU where only a fraction of experts fit, a global ranking starves cold layers (left heatmap), while a per-layer policy maintains coverage at every layer (right heatmap). Per-layer caching enables topologies that a flat global ranking cannot express. The mechanical payoff is PCIe stall elimination: expert offloading is memory-bandwidth-bound, so every cache miss costs a CPU-to-GPU transfer. When per-layer caches cover each layer's working set, steady-state misses drop to zero, which is why a hit-rate improvement turns into a 6.4x wall-clock gain (10.22 vs 1.60 tok/s on DeepSeek-V2-Lite at matched total budget).

3. The allocation signal does not matter. We tested six signals (Shannon entropy, inverse top-k mass, inverse variance, inverse KL, inverse Gini, uniform). None differentiates from uniform by more than 2.5pp in hit rate, and all six collapse to within noise of uniform in wall-clock on two models. Uniform is the default. Shannon entropy is opt-in for models with high inter-layer entropy spread (ΔH around 1 nat or more), but it was within noise of uniform on every model tested end-to-end.

Strategy	Total slots	Hit Rate	Δ vs shared	Wall-clock (A100)
Shared cache	32	48.6%	baseline	1.60 tok/s
Per-layer uniform	864 (27x)	63.3%	+14.7pp	10.22 tok/s
Per-layer entropy	864 (27x)	65.5%	+16.9pp	10.17 tok/s (~ uniform)

---

Mechanism vs policy: Fiddler head-to-head (A100-80GB, Mixtral-8x7B)

Fiddler and MoE-PolicyLang on the same hardware, model, prompt, and methodology (n=5, greedy decoding, 64 tokens):

Config	tok/s (±σ)	95% CI	GPU Peak	Hit Rate	Transfers
Fiddler	4.17 ± 0.02	[4.16, 4.18]	80.6 GB	88.3%	—
MPL fiddler_equiv (cap=2)	0.18 ± 0.00	[0.18, 0.18]	6.4 GB	19.5%	4,283
MPL balanced (cap=4)	0.29 ± 0.00	[0.29, 0.29]	39.5 GB	46.4%	2,665
MPL generous (cap=6)	0.45 ± 0.00	[0.45, 0.45]	61.7 GB	71.0%	1,726

All MPL configs produce bit-identical output. Fiddler is 9-23x faster.

The gap is mechanism, not policy. Fiddler uses an optimized C++/CUDA transfer pipeline with pre-allocated GPU memory slots and direct DMA. MoE-PolicyLang dispatches through Python-level Tensor.to() calls in the HuggingFace forward pass. At Fiddler's 85% GPU residency (217/256 experts on-device), placement strategy isn't doing the work; the model mostly fits.

MoE-PolicyLang is a policy specification layer, not a serving system. It specifies which experts to cache, evict, and prefetch, but does not implement the physical mechanism that moves expert tensors between devices.

vLLM backend: same policy, different mechanism

A second backend integrates with vLLM for production-grade quantized MoE inference. VLLMPolicyRunner instruments vLLM's router layers to capture expert routing decisions and feeds them through the same DSL, compiler, and hooks that the HuggingFace backend uses.

from moe_policylang.integrations.vllm_backend import VLLMPolicyRunner

runner = VLLMPolicyRunner(
    model="Qwen/Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4",
    policy_dsl='''
        policy demo {
            cache { capacity = 8  eviction = lru }
            prefetch { strategy = lookahead  lookahead = 1 }
            schedule { mode = gpu_only }
        }
    ''',
    quantization="gptq",
)

results = runner.generate(["What is expert routing?"], max_tokens=30)
print(results["text"])          # generated text
print(results["policy_stats"])  # cache hits, prefetch accuracy, etc.

Verified on RTX 5080 (16 GB), vLLM 0.21, GPTQ-Int4 quantization. Captures 744 routing events across 24 layers × 60 experts, with a 14.7% cache hit rate and 72% prefetch accuracy from a minimal 8-slot LRU policy.

The same .policy policy text runs on both backends; only the mechanism layer (HF eager Python vs vLLM optimized kernels) differs. Combined with the Fiddler head-to-head above, this is the empirical case for the policy/mechanism separation — the abstraction holds across two production mechanism layers, not just structurally.

---

Cache hit rates: capacity sweeps

Capacity sweeps on offline traces:

• Mixtral-8x7B (8 experts, top-2) saturates at cap=8 with around 100% hit rate, since all experts fit. Policy choice barely matters here.
• DeepSeek-V2-Lite (64 experts, top-6) reaches only 51% hit rate at cap=32 (half the experts). LFU consistently beats LRU across budgets because DeepSeek has significant frequency skew (some experts activated 3-5x more often). This is the regime where policy selection and per-layer budgeting make a measurable difference.

---

Policy authoring effort

To add a new policy variant to one of these systems, a developer needs to read and modify the system's expert-management module. MoE-PolicyLang replaces that with a short .policy file. The 14–40x numbers below count lines a user writes to express a policy; they do not include MoE-PolicyLang's own runtime, which is around 4,300 LOC.

System	Expert-mgmt module	DSL equivalent	Authoring reduction
Fiddler	280 LOC	7 lines	40x
HybriMoE	~500 LOC	14 lines	36x
MoE-Infinity	520 LOC	16 lines	33x
vLLM	300 LOC	12 lines	25x
ExpertFlow	~400 LOC	16 lines	25x
FineMoE	~350 LOC	25 lines	14x

Methodology: non-blank, non-comment lines in the primary expert-management module. Measured sources: Fiddler from set_expert_loc() + execute_fiddler() in src/fiddler/mixtral.py (280 LOC); MoE-Infinity from expert_prefetcher.py + expert_cache.py (520 LOC); vLLM from MixtralMoE expert dispatch in vllm/model_executor/ (300 LOC). Counts marked ~ are estimated from paper descriptions of closed-source systems. Switching between strategies (LRU to LFU) is a one-word change in the DSL versus rewriting cache data structures in the hand-coded version.

---

Dispatch overhead

Per-layer dispatch (the Python hook that decides cache/evict/prefetch) adds under 3.2% of MoE forward-pass time on A100: 6–47 µs/layer against a 1,459 µs baseline. This is the policy decision overhead; cache misses and weight transfers are accounted for separately and depend on the policy and workload.

---

Installation

From PyPI:

pip install moe-policylang           # DSL only (no GPU deps)
pip install moe-policylang[gpu]      # + torch, transformers, accelerate
pip install moe-policylang[vllm]     # + vLLM (GPTQ/AWQ quantized inference)
pip install moe-policylang[all]      # everything

For quantized models, use the [vllm] extra. vLLM handles GPTQ and AWQ quantization with optimized kernels; MoE-PolicyLang observes routing decisions and applies policy logic without managing the tensors directly.

For Blackwell GPUs (RTX 5080/5090), set these env vars before running vLLM:

export VLLM_USE_FLASHINFER_SAMPLER=0
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export VLLM_FLASH_ATTN_VERSION=2

From source (development):

git clone https://github.com/jesse-pokora/MoE-PolicyLang.git
cd MoE-PolicyLang
pip install -e ".[dev,gpu]"

Cython fast path (for complex policies):

pip install moe-policylang[cython]
python setup_cython.py build_ext --inplace

Python dispatch ranges from 6 µs/layer (simple LRU) to 47 µs/layer (composed policies with triggers). The Cython path targets the high end: freq_threshold and composed_full drop from 28-47 µs to under 10 µs/layer. Simple policies like lru_basic (6 µs) see no benefit.

---

Tested Models

MoE-PolicyLang auto-detects MoE structure from any HuggingFace model with no model-specific code required. Evaluated on:

Model	Experts x Layers	Routing	Hardware	Backend
Mixtral-8x7B-Instruct	8 x 32	top-2	A100-80 GB	HF Transformers
DeepSeek-V2-Lite	64 x 27	top-6	A100-80 GB	HF Transformers
Qwen1.5-MoE-A2.7B	60 x 24	top-4	RTX 5080 (16 GB)	HF Transformers
Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4	60 x 24	top-4	RTX 5080 (16 GB)	vLLM
OLMoE-1B-7B	64 x 16	top-8	RTX 5080 (16 GB)	HF Transformers

---

Project Structure

moe_policylang/
├── grammar.lark           # Lark LALR grammar (62 productions)
├── parser.py              # Grammar → PolicyIR
├── ir.py                  # Intermediate representation
├── validator.py           # 20 semantic validation rules
├── compiler.py            # IR → CompiledPolicy
├── auto.py                # Auto-generate policies from model + GPU
├── dsl.py                 # Python eDSL (@sched.policy decorator)
├── adaptive.py            # Adaptive policies (adapt blocks)
├── autotuner.py           # Grid-search policy optimizer
├── cli.py                 # CLI: validate, compile, run
├── runtime/
│   ├── hooks.py           # 5-step per-layer dispatch protocol
│   ├── cache.py           # LRU / LFU / Score / FreqThreshold
│   ├── prefetch.py        # Affinity / History / Lookahead
│   ├── scheduler.py       # GPU-only / CPU-fallback / Hybrid
│   ├── per_layer.py       # PLCB — per-layer cache budgeting
│   ├── triggers.py        # Memory-pressure & TTL eviction
│   └── _fast/             # Cython-accelerated paths
└── integrations/
    ├── __init__.py         # attach() — main user API
    ├── huggingface.py      # HuggingFace Transformers hooks
    ├── vllm_backend.py     # vLLM integration (routing trace + policy replay)
    ├── weight_placement.py # Expert offloading manager
    └── async_transfer.py   # CUDA stream async transfers

---

Running Experiments

# Offline trace replay (no GPU needed)
python scripts/run_eval.py
python scripts/run_sweep.py
python scripts/run_diagnostics_demo.py   # v1.5.3 routing + replay walkthrough

# Live inference on consumer GPU
python scripts/run_dsl_demo.py
python scripts/run_constrained_e2e.py

# Generate all paper figures
python scripts/generate_figures.py

# Benchmarks & evaluations (requires CUDA GPU + model weights)
python scripts/bench_qwen_multirun.py   # Qwen throughput (Table 4)
python scripts/bench_coldstart.py       # Cold-start throughput analysis
python scripts/bench_power.py           # Power/energy measurement
python scripts/eval_quality.py          # Perplexity evaluation (wikitext-2)
python scripts/ablation_plcb_sensitivity.py  # PLCB hyperparameter sweep
python scripts/plot_coldstart.py        # Generate cold-start figure

---

Tests

python -m pytest tests/ -q

490+ tests covering parsing, validation, compilation, runtime dispatch, adaptive policies, per-layer PLCB, routing diagnostics, offline trace replay, and integration hooks.

---

Routing diagnostics & offline replay (v1.5.3)

Three APIs help you decide whether a policy is architecturally compatible with a given MoE model and benchmark new policies offline:

from moe_policylang.integrations.accessors import auto_accessor
from moe_policylang import validate_policy, parse_policy
from moe_policylang.benchmark.replay import replay_policies

# 1. Static fingerprint (zero GPU): is this model flat- or sharp-routing?
accessor = auto_accessor(model)
print(accessor.routing_profile().concentration_class)   # 'flat-likely' / 'mixed' / 'sharp-likely'

# 2. Validator warnings: catch policy/architecture mismatches before deploy.
ir = parse_policy(open("my_policy.policy").read())
for w in validate_policy(ir, accessor):                 # backward-compat: drop accessor for old behavior
    print("WARN:", w)

# 3. Offline benchmark: sweep 8 cache policies against a recorded trace.
results = replay_policies("traces/olmoe.parquet", top_k=8,
                          policies=["topk_lru", "score_threshold", "belady"],
                          budget_fractions=[0.25, 0.50, 0.75])

See docs/RESEARCH.md for the full workflow guide (recorder → persistence → policy sweep → topology sweep, plus how to plug in a new prefetch policy).

---

What's new in 1.12.0 — Serving engine v0.2: per-expert control mode

v1.11 shipped the observer-mode executor; v1.12 promotes it to a real policy executor. When the policy says skip an expert, the expert's forward is never called — its contribution is dropped or the remaining experts are renormalized. This is the first time the toolkit actually executes a policy decision against live weights.

from moe_policylang.serving import PathBExecutor

executor = PathBExecutor(
    model_id="allenai/OLMoE-1B-7B-0924",
    policy_path="examples/hobbit_weighted_sum.policy",
    control_mode="per_expert",      # v0.2 — actually swap experts
    skip_strategy="drop",            # "drop" | "renormalize"
)
stats = executor.run(["Explain QED."], max_new_tokens=64, compute_perplexity=True)
print(stats.summary_line())
print(f"skipped {stats.experts_skipped_total} / "
      f"{stats.experts_skipped_total + stats.experts_evaluated_total} "
      f"expert calls ({stats.skipped_fraction:.1%}) — PPL={stats.mean_perplexity:.2f}")

How v0.2 works

1. find_moe_blocks(model) walks the model and locates each MoE block matching the (gate + experts nn.ModuleList) pattern. Works on OLMoE / Qwen1.5-MoE / JetMoE / Granite-MoE. (Mixtral on transformers 5.x uses fused expert tensors — defer to v0.3.)
2. install_per_expert_control() replaces each block's forward with a controlled version that consults the PolicyHook before dispatching to experts.
3. For each token, the controller asks the hook for a DispatchPlan and routes only to experts in evaluated_experts; experts in skipped_experts are not called.
4. On uninstall, the original forwards are restored byte-for-byte (via a saved reference per block).

Skip strategies

Strategy	Behavior	When to use
drop (default)	Skipped expert contributes 0 to the layer output	Captures the isolated quality cost of skipping that expert
renormalize	Remaining experts' router scores re-softmaxed	Output magnitude is preserved; matches the routing semantics of models that use norm_topk_prob=True

What's measurable now

Stat	What it tells you
experts_skipped_total	Total per-(token, layer) skip count across the run
experts_evaluated_total	Total per-(token, layer) evaluate count
skipped_fraction	skipped / (skipped + evaluated)
control_decisions	Per-token decision log: layer, selected, evaluated, skipped
per_prompt_perplexity	Quality cost of the skips, end-to-end

Compare a policy's mean_perplexity under control_mode='per_expert' to its mean_perplexity under control_mode='observer' (vanilla inference) to get the quality cost in nats per token.

Test surface

12 new tests in tests/test_serving_control.py exercise the structural detection, install/restore round-trip, both skip strategies, state counters, and PathBExecutor integration.

1006 tests passing. PyPI: pip install moe-policylang==1.12.0.

---

What's new in 1.11.0 — Path B serving engine (observer mode)

v1.11 adds moe_policylang.serving, the HF-wrapped policy executor that runs audit-recommended policies on live HuggingFace MoE models and measures actual wall-clock + perplexity vs the vanilla baseline.

Previously the toolkit only had the simulator (estimated wall-clock via per-tier latency constants) and VLLMPolicyRunner (observation of vLLM router events). The serving submodule is the first piece of the stack that produces measured numbers from real GPU inference.

from moe_policylang.serving import PathBExecutor, run_benchmark

# Single-policy run
executor = PathBExecutor(
    model_id="mistralai/Mixtral-8x7B-Instruct-v0.1",
    policy_path="examples/hobbit_weighted_sum.policy",
    device_map="auto",
)
stats = executor.run(
    prompts=["Explain QED in one paragraph."],
    max_new_tokens=64,
    compute_perplexity=True,
)
print(stats.summary_line())
# policy=hobbit_repro model=mistralai/... tok/s=42.3 PPL=8.71 events=128

# N-policy comparison
result = run_benchmark(
    model_id="mistralai/Mixtral-8x7B-Instruct-v0.1",
    policy_paths=[
        "examples/topk_lru_baseline.policy",
        "examples/hobbit_weighted_sum.policy",
        "examples/deepseek_moe_shared_experts.policy",
    ],
    prompts=["Explain QED.", "Write fibonacci.", "Translate hello to French."],
    max_new_tokens=64,
)
print(result.render())
print(f"best: {result.best_by('throughput').policy_name}")

Module layout

moe_policylang.serving/
  executor.py     — PathBExecutor (v0.1 observer mode)
  hf_wrap.py      — HF MoE block forward-hook interception
  timing.py       — CUDA-event recorder + per-token CE for perplexity
  benchmark.py    — N-policy comparison harness

v0.x staging

Version	Capability
v0.1 (this release)	Observer: records policy decisions + measures live wall-clock + perplexity vs vanilla. No actual expert skipping yet.
v0.2	Per-expert ModuleList control (skip/swap experts on transformers <5.x)
v0.3	Fused MoE block (transformers 5.x Mixtral)
v0.4	Multi-GPU + EP/TP
v0.5	KV-cache integration (T2.5 strict mode)

Why public

Pure-Python HF-wrapped policy execution isn't a moat — anyone can write it. The serving submodule stays public alongside the rest of the toolkit while it's pure Python. Custom CUDA / C++ kernels (when we eventually write them) move to the reserved private repo jesse-pokora/moe-serving. See decisions/serving-engine-design.md for the architecture decisions.

Test surface

19 new tests in tests/test_serving_*.py use a stub-MoE fixture that runs on CPU — no GPU or HF model download needed for the test suite. The real-Mixtral demo lives in examples/serving_benchmark_mixtral.py for GPU runs.

994 tests passing. PyPI: pip install moe-policylang==1.11.0.

---

What's new in 1.10.0 — Sprint A toolkit polish

v1.10 closes five gaps the v1.9 audit pack surfaced: richer trace schema, DSL extensions, simulator integrations, HOBBIT-faithful composite eviction, and a paper-ready report generator. See router-oracle/decisions/v2-roadmap.md for the full sprint plan.

Trace schema extensions (Sprint A item 1)

7 new optional TraceEntry fields, each unlocks an audit module's strict mode automatically:

Field	Unlocks
token_id	affinity_table, token_id_routing
phase (prefill|decode)	phase_split
language_tag	multilingual_alignment
domain_tag	domain_specialization
router_logits (pre-softmax)	counterfactual_routing
token_loss (per-token CE)	counterfactual_routing
kv_shard_id	kv_cache_locality

Collect via record_trace(record_token_ids=True, record_phase=True, ...) or python phase1_collect_traces.py --capture-all.

DSL extensions (Sprint A item 2)

Four new surfaces, each closing an audit ↔ DSL gap:

hierarchy {
    tier T_shared { dtype = fp16 location = gpu capacity = 2 residency = always }   # DeepSeekMoE
    tier T0_hot   { dtype = fp16 location = gpu capacity = 4
                    dtype_ladder = [fp16, int8, int4, int2] }                       # Q-MoE / MoQE
    tier T1_warm  { dtype = int4 location = gpu capacity = 16 }
    granularity_split = 4                                                            # DeepSeek-V2 split
    eviction = lru
}
cache { capacity = 8 capacity_factor = 1.25 eviction = lru }                        # Switch / GShard

Runtime semantics: residency = always makes a tier eviction-immune (modeling DeepSeekMoE always-on shared experts). TierController auto-rejects new inserts when an ALWAYS tier is full.

Simulator integrations (Sprint A item 3)

TierController(hierarchy, compute_us_per_layer=200.0)     # Comm-compute overlap
TierController(hierarchy, eviction_backend='countmin')    # Sketched LFU backend
TierController(hierarchy, asymmetric_latency=True)        # Cold/warm/dequant-start latency
TierController(hierarchy, sketch_membership=True)         # Bloom-filter membership

New tc.overlapped_wall_us property — per-layer max(compute_us, sum_of_fetches) — mirrors audit.overlap numbers on the live simulator.

HOBBIT-faithful weighted_sum (Sprint A item 4)

The weighted_sum(...) composite-eviction policy now has real LHU

• FLD implementations (no more LRU fallback for those components):

eviction = weighted_sum(lru = 0.3, lfu = 0.3, lhu = 0.2, fld = 0.2)

• LHU (Least-Hot-Used): low count in the recent sliding window (default 256 accesses)
• FLD (Frequency-Length-Distance): low hit_count / age ratio

Both also exposed as standalone enum values (eviction = lhu, eviction = fld).

moe-policylang report <trace> (Sprint A item 5)

Single command, paper-ready markdown:

moe-policylang report traces/olmoe/chat/trace.parquet \
    --output report.md --capacity 0.25 --budget 16

Combines audit fingerprint (core + extended) + cache-policy recommendation + mixed-precision recommendation + quant ladder + pruning candidates into one document. See notebooks/audit_pack_walkthrough.ipynb for the equivalent Python flow.

973 tests passing. PyPI: pip install moe-policylang==1.10.0.

---

What's new in 1.9.0 — Audit pack expansion: efficiency hypotheses + Tier 1/2/3 + axes

v1.9 takes the v1.8 4-module audit pack to 24 audit surfaces — seven efficiency hypotheses, six new Tier 1 modules, four Tier 3 simulation modules, and eight Tier 2 modules (six with realized proxy modes that work on the v1 trace schema, two stubs that activate when richer columns are present). Plus six missing- axis design docs that scope what's still beyond the trace-replay model.

moe-policylang audit traces/mixtral/chat/trace.parquet --extended

# Fingerprint — traces/mixtral/chat/trace.parquet
  geometry: 246 tokens × 32 layers × 8 experts, top_k=2
  coactivation: layers=32, experts=8, k=2, mean_max_lift=1.78
  cross_layer (adjacent pairs): mean_MI=0.426 nats, mean_colocation_savings=18.5%
  layer_redundancy: layers=32, mean=0.140, depth_gradient=+0.0000/layer
  sequence_budget: sequences=1, top_k=2, mean_slack=18.996 mass, slack_fraction=0.8% of token-reachable
  --- v1.9 extended ---
  speculative_topk: ε=1: cov=24.0%, overhead=1.50×, ε=2: cov=37.7%, ε=4: cov=67.4%
  position_routing: bins=8, entropy_spread=0.000, max_TV_from_global=0.001
  batch_active_set: B=1: union=12%, gini=0.00, B=32: union=94%, gini=0.01
  layer_skipping: mean_skipworthy=82.8%
  margin_sensitivity: margin p50=0.005, p10=0.001, p01=0.001, flip(@ε=0.05)=86.7%
  capacity_factor: cf=0.5: drop=22.7%, cf=1.0: drop=11.8%, cf=2.0: drop=0.0%
  shared_expert: 0/8 experts cover ≥80% of tokens
  prediction_ceiling(L0→L1): top1=37.1%, top-k=39.7%

What it audits (24 modules)

Efficiency hypotheses (7) — measurable opportunities for runtime speedup, each tied to a specific 2024-26 paper or implementation:

Module	Audits
affinity_table	Cache token→expert decisions (Cache-Conditional Experts, ICLR 2025)
speculative_topk	Load top-(k+ε) to reduce demand misses
runtime.asymmetric_latency	Cold/warm/dequant-start latency model (opt-in on TierController)
overlap	Comm-compute overlap potential (standard pipeline hiding)
router_decouple	CPU-side router lookahead (RouterPrep-style)
quant_ladder	fp16→int8→int4→int2 per-expert ladder (Q-MoE, MoQE, MC-MoE)
runtime.sketch_cache	Bloom + count-min sketch cache backends (opt-in)

Tier 1 audit modules (6) — measurable from existing trace schema:

Module	Audits
mixed_precision	Per-expert hot/cold dtype split recommendation
pruning_candidates	N least-accessed experts droppable at ≤X% routing change
position_routing	Per-position-bin routing entropy + TV from global
batch_active_set	Active-expert union saturation curve vs batch size (Diff-MoE)
layer_skipping	Skipworthy-token fraction per layer (MoE++, MoDES)
margin_sensitivity	Router fragility under ε-perturbation (When Routing Collapses)

Tier 3 simulation modules (4) — modelling extensions:

Module	Audits
capacity_factor	Token-drop rate sweep at varying capacity (Switch / GShard)
shared_expert	Always-on shared-expert candidate detection (DeepSeekMoE)
granularity	Fine-grained vs coarse-grained synthesis (DeepSeek-V2 split argument)
prediction_ceiling	Online next-layer expert predictability (SP-MoE, DAOP)

Tier 2 modules with realized proxy modes (3 — graduate to strict mode when richer trace columns are present):

Module	Proxy mode	Strict mode requires
counterfactual_routing	margin + alternative-set surrogate	router_logits + token_loss
kv_cache_locality	simulated round-robin placement	kv_shard_id + placement map
mtp_prefetch	learned next-layer marginal predictor	paired (main, MTP-draft) traces
phase_split	first-N-tokens-as-prefill proxy	phase column
token_id_routing	position-as-proxy R²	token_id column

Tier 2 modules still data-gated (3 — design + dataclass shipped):

Module	Activates when...
multilingual_alignment	language_tag column present
domain_specialization	domain_tag column present
quant_stability	paired fp16 + int4 traces present

Missing-axes design docs (6) — entire measurement axes scoped but not implemented: training dynamics, hardware utilization, quality feedback, newer architecture variants, multi-tenant, fairness. See decisions/missing-axes/.

Runtime extensions (opt-in)

from moe_policylang.runtime.tier_controller import TierController

# Cold/warm/dequant-start latency model
tc = TierController(hierarchy, asymmetric_latency=True)

# Bloom-filter-backed cache membership state (memory savings on bookkeeping)
tc = TierController(hierarchy, sketch_membership=True, sketch_membership_fpr=0.01)

Programmatic API

import moe_policylang as mpl
from moe_policylang.audit import audit_trace

fp = audit_trace("traces/olmoe/chat/trace.parquet", extended=True)
print(fp.render())
# render() includes the v1.9 extended block

n=3 paper-ready findings (v1.9 expansion)

Run on existing OLMoE / Qwen1.5-MoE / Mixtral trace corpus (router-oracle/results/tables/audit-n3-fingerprints.md):

Architecture	margin p10	flip(@ε=0.05)	drop @cf=1.0	mean skipworthy
OLMoE	~3e-6	~100%	0.0%	99.9%
Mixtral	~0.001	86-87%	11-22%	~80%
Qwen	~1e-6	~100%	0.0%	99.9%+

Four publishable v1.9 findings:

1. Margin fragility is universal: every architecture flips ≥86% of routing decisions under ε=0.05 perturbation; OLMoE and Qwen flip ~100% (intrinsic router fragility, not just adversarial).
2. Mixtral hits capacity limits: 22% of tokens dropped at cf=0.5, 12% at cf=1.0. OLMoE and Qwen have zero drops because their flat routing keeps per-expert load well under capacity.
3. Skipworthy-token fraction is huge on flat-likely routers: OLMoE/Qwen show 99.9% of tokens have top-1 mass <0.3 (the router is "uncertain" on essentially every token), suggesting most layers add little signal.
4. Single-layer prediction ceiling sits at 30-67% top-k accuracy on n=3: well above chance (~12-25%) but below the 84-91% ceiling claimed in MoE-SpeQ/SP-MoE — re-collection with richer features required to close that gap.

Remote-GPU audits

Companion router-oracle/colab/ ships a Colab remote-control workflow for models too big for local hardware (DeepSeek-V2-Lite, Qwen3-MoE, etc.):

python tools/launch_colab.py --name <id> --kind audit \
    --model deepseek-ai/DeepSeek-V2-Lite --workload chat

912 tests passing. PyPI: pip install moe-policylang==1.9.0.

---

What's new in 1.8.0 — Architecture-claim audit pack

A focused new package, moe_policylang.audit, that quantifies the architectural claims published 2024-2026 against any HuggingFace MoE trace. Four modules + an orchestrator + a CLI command:

moe-policylang audit traces/mixtral/chat/trace.parquet

# Fingerprint — traces/mixtral/chat/trace.parquet
  geometry: 246 tokens × 32 layers × 8 experts, top_k=2
  coactivation: layers=32, experts=8, k=2, mean_max_lift=1.78
  cross_layer (adjacent pairs): mean_MI=0.426 nats, mean_colocation_savings=18.5%
  layer_redundancy: layers=32, mean=0.140, depth_gradient=+0.0000/layer
  sequence_budget: sequences=1, top_k=2, mean_slack=18.996 mass, slack_fraction=0.8% of token-reachable
  mass_gini: 0.003, chosen_gini: 0.023, mean_topk_mass: 0.309

# Top co-activating pairs (per layer)
  layer 0: (0,1)=1.39x, (2,3)=1.17x, (1,4)=1.09x
  layer 1: (5,7)=1.18x, (1,2)=1.02x, (2,4)=0.98x
  layer 2: (2,4)=1.24x, (0,7)=1.06x, (1,5)=0.97x

What it measures

Module	Claim it audits	Headline literature	Status
coactivation (T1.1)	Within-layer expert pair correlation; informs prefetch/colocation	Pre-gated MoE; MoE-Infinity	Empirical co-activation lift per pair, per layer
cross_layer (T1.2)	Cross-layer Markov structure (P(e_b	e_a)); informs placement	ExFlow (claims up to 67% routing-latency reduction)
layer_redundancy (T1.6)	Deeper layers carry more redundancy than shallow ones; global pruning beats uniform	MoE-I², NAEE, REAP, MoE Pathfinder	Routing-based redundancy proxy + suggested layer-wise prune budgets
sequence_budget (T1.11)	Token-wise top-k wastes budget; sequence-level reallocation wins	Route Expert by Sequence (claims up to 16.89% gain under high sparsity)	Token vs sequence reachable-mass slack

Programmatic API

import moe_policylang as mpl
from moe_policylang.audit import audit_trace

fp = audit_trace("traces/olmoe/chat/trace.parquet")
print(fp.render())

# Individual sub-modules
from moe_policylang.audit import (
    coactivation_matrix, cross_layer_affinity,
    per_layer_redundancy, sequence_budget_slack,
)

n=3 paper-table from the audit pack

Run on our existing OLMoE / Qwen1.5-MoE / Mixtral trace corpus (router-oracle/results/tables/audit-n3-fingerprints.md):

Architecture	max-lift	cross-layer MI	colocation savings	seq-slack
OLMoE	53.0×	1.76 nats	32.8%	0.0%
Qwen	6.75×	2.41 nats	31.4%	0.1%
Mixtral	1.70×	0.43 nats	17.5%	0.9%

Three findings:

1. Co-activation lift varies by ~30× across architecture classes.
2. ExFlow's 67% colocation-savings claim is validated as an upper bound; typical workloads see 14-46%.
3. Sequence-level reallocation slack is < 1% on every cell — SeqTopK's gains do not reproduce on natural workloads at top_k ≥ 2.

Remote-GPU audits

The router-oracle companion repo ships a Colab remote-control workflow for running audits on models too big for local hardware (DeepSeek-V3, Qwen3-MoE, etc.):

python tools/launch_colab.py \
    --name deepseek_v3_audit_chat \
    --kind audit \
    --model deepseek-ai/DeepSeek-V3 \
    --workload chat

The launcher commits a job spec to the repo and prints a Colab URL. See router-oracle/colab/README.md for the full flow.

883 tests passing. PyPI: pip install moe-policylang==1.8.0.

---

What's new in 1.7.0 — Per-layer hierarchies + causal analysis + contextual bandit

Four additions that round out the production-tuning story for hierarchies:

import moe_policylang as mpl
import numpy as np

1. Per-layer hierarchies — different tier configs per layer band:

policy mixtral_banded {
    per_layer_hierarchy {
        band layers = 0..7 {
            tier T0_hot  { dtype = fp16  location = gpu  capacity = 4 }
            tier T1_warm { dtype = int4  location = gpu  capacity = 12 }
            eviction = lru
        }
        band layers = 8..31 {
            tier T0_hot  { dtype = fp16  location = gpu  capacity = 2 }
            tier T1_warm { dtype = int4  location = gpu  capacity = 8 }
            tier T2_cold { dtype = int4  location = cpu  capacity = 64 }
            eviction = lfu
        }
    }
}

Runtime backing is LayeredTierController — routes each access to the right band's controller, aggregates stats across bands. Bands must be non-overlapping (parser-enforced).

2. Extended adaptive metrics — adapt { ... } blocks now accept drift, utilization_gini, dead_expert_rate, cost_per_token, and cold_promotion_rate in addition to hit_rate / eviction_rate. Pluggable via MetricRegistry. Adapt rules now work on hierarchy {} policies (previously only cache {} policies).

policy adaptive_drift {
    hierarchy {
        tier T0 { dtype = fp16 location = gpu capacity = 8 }
        tier T1 { dtype = int4 location = cpu capacity = 32 }
        eviction = lru
    }
    adapt {
        when drift > 0.2 for 200 accesses { eviction = lfu }
        when utilization_gini > 0.85 { eviction = score }
    }
}

3. Causal analysis via synthetic traces — controllable knobs (concentration / autocorr / tail mass / hot-set fraction), then sweep one knob at a time to verify the selector's recommendation tracks each property in isolation:

report = mpl.causal_audit(num_experts=32, top_k=4, capacity_fraction=0.25)
print(report.summary())
# # Causal audit
# ## concentration
# - match rate: 80%  (4/5)
# - mean regret: 0.42 pp
# ## autocorr
# - match rate: 100% (5/5)
# - mean regret: 0.00 pp
# ## tail_mass
# - match rate: 80%  (4/5)
# - mean regret: 0.31 pp

4. RL-based online policy adaptation — LinUCB contextual bandit over a finite set of policy arms; learns the (routing-profile-features → best policy) mapping online.

bandit = mpl.LinUCBPolicyBandit(
    arms=["topk_lru", "score_threshold", "score_threshold_p95", "rank_prefetch"],
    context_dim=5, alpha=1.0,
)
for window_softmax in stream_router_outputs():
    ctx = mpl.default_context_from_softmax(window_softmax, top_k=4)
    chosen = bandit.decide(ctx)
    hit_rate = run_window_with_policy(chosen.arm)   # your replay loop
    bandit.observe(chosen.arm, hit_rate)

Use the bandit when the cache-selector's rule-based recommendation isn't stable for your workload mix; skip it when the selector already gives you a clear winner (the bandit adds ~4-7 pp regret over the first ~50 windows while it explores).

864 tests passing. PyPI: pip install moe-policylang==1.7.1.

ContextualBanditAdaptiveHook (1.7.1) is the drop-in adapter that wraps any PolicyHook and drives a LinUCBPolicyBandit from per- window stats, swapping the underlying eviction policy as the bandit explores.

---

What's new in 1.6.0 — Atlas + Selector + Synthesizer + Detector

Five new top-level tools that turn the v1.5.x DSL + replay surface into a full policy-recommendation pipeline:

import moe_policylang as mpl

# 1. SELECT a policy for a given model architecture
profile = mpl.auto_accessor(model).routing_profile()
rec = mpl.select_policy(profile, capacity_fraction=0.25)
print(rec.policy, rec.rationale)
# > score_threshold_p95
# > flat-likely router; high-confidence threshold dominates LRU

# 2. SYNTHESIZE a new policy by searching the DSL grammar space
r = mpl.synthesize_policy(
    trace_path="traces/olmoe/code/trace.parquet",
    top_k=8, num_experts=64,
    memory_budget_experts=16.0,
    num_iterations=100, method="genetic",
)
print(r.policy_dsl)        # parseable .policy file
print(r.hit_rate, r.belady_gap)

# 3. DETECT adversarial routing patterns online
det = mpl.is_adversarial(softmax_window)
if det.is_adversarial:
    print(f"Adversarial input (class={det.suspected_class}, conf={det.confidence:.2f})")

# 4. DETECT distribution drift over time
detector = mpl.DriftDetector(baseline_window=128, drift_threshold=0.20)
for row in router_softmax_stream:
    report = detector.update(row)
    if report.drifted:
        re_select_policy()      # workload has shifted

# 5. Export Prometheus metrics from a TierController
from moe_policylang.runtime.prometheus_emitter import PrometheusEmitter
emitter = PrometheusEmitter(tier_controller, policy_name="prod")
emitter.emit()       # call periodically; metrics scrapeable at /metrics

DSL extensions in 1.6.0:

• Composite eviction: eviction = weighted_sum(lru=0.4, lfu=0.6) — HOBBIT-style multi-signal eviction
• Multi-GPU tiers: tier T0 { location = gpu:0 capacity = 8 } — pin tiers to specific accelerators for expert-parallel serving

New CLI (8 subcommands):

moe-policylang simulate trace.parquet policy.policy
moe-policylang select model_config.json --capacity 0.25 --atlas atlas.csv
moe-policylang synthesize trace.parquet --budget 16 --method genetic
moe-policylang detect trace.parquet --window 32
moe-policylang explain policy.policy trace.parquet
moe-policylang cost policy.policy trace.parquet --gpu-rate-usd-per-hour 1.50

Other:

• .policy is now the canonical extension (.moe retired; parse_file rejects it).
• pip install moe-policylang now correctly pulls torch + numpy + pyarrow as hard dependencies.
• Docker image: docker build -t moe-policylang:1.6.0 . + docker run --rm moe-policylang:1.6.0 version.
• ReadTheDocs scaffold + Sphinx config under docs/.
• See BENCHMARK.md for the full benchmark recipe and COMMERCIAL.md for an honest commercial-viability analysis.

---

Documentation

The two manuals are listed near the top under Manuals: docs/MANUAL.md (DSL reference) and docs/RESEARCH.md (research workflow).

---

Glossary

• MoE (Mixture-of-Experts). A Transformer layer that replaces a single feed-forward block with N expert networks plus a small router that picks the top-k experts per token.
• Expert. One feed-forward sub-network inside an MoE layer. Mixtral has 8 per layer, Qwen1.5-MoE has 60, DeepSeek-V2-Lite has 64.
• Router / top-k routing. The small classifier inside each MoE layer that scores experts and picks the k highest per token.
• Offloading. Keeping some weights in CPU RAM and moving them to GPU on demand. The cost is the PCIe transfer.
• PCIe. The bus between CPU memory and the GPU. Roughly two orders of magnitude slower than reading from GPU HBM/GDDR, so cache misses are expensive.
• Skeleton. Everything in the model that isn't an expert: embeddings, attention, layer norms, LM head. MoE-PolicyLang pins the skeleton on GPU (≈3.7 GB for Qwen1.5-MoE) and only the expert weights move.
• Cache hit / miss. A hit means the expert the router picked is already on GPU. A miss means we have to fetch it (or run it on CPU, if schedule = cpu_fallback).
• LRU / LFU. Least-Recently-Used and Least-Frequently-Used cache eviction. LRU drops whatever hasn't been touched lately; LFU drops whatever has the lowest activation count (with a decay factor so old hot experts age out).
• fp16 / GPTQ / AWQ. Weight precisions. fp16 is the standard half-precision format used in this paper's experiments. GPTQ and AWQ are 4-bit quantization formats that vLLM consumes; they trade a small amount of perplexity for a large VRAM reduction.
• KV-cache. The cache of attention keys/values from previous tokens during generation. It grows with sequence length and competes with expert weights for VRAM.
• pp (percentage points). Used for hit-rate deltas: +14.7 pp means 48.6% → 63.3%, not a 14.7% relative change.
• PLCB. Per-Layer Cache Budgeting. See the section above; the load-bearing part is the per-layer cache structure, not the entropy allocator that gave the technique its name.

---

Citation

@misc{pokora2026moepolicylang,
  title={MoE-PolicyLang: A Domain-Specific Language for Mixture-of-Experts Scheduling Policies},
  author={Pokora, Jesse},
  year={2026},
  url={https://github.com/jesse-pokora/MoE-PolicyLang}
}

---

License

MIT License — Copyright (c) 2026 Jesse Pokora