experts4bit-qlora

Train and serve fused Mixture-of-Experts models in 4-bit on hardware that cannot hold them in bf16.

The problem in one line: load_in_4bit=True leaves a fused MoE's expert weights in bf16, so the model still OOMs; this package quantises exactly those experts, fine-tunes them with QLoRA, keeps them in host RAM or on NVMe when they do not fit, and serves them on one consumer NVIDIA GPU. Canonical package: experts4bit-qlora on PyPI (import experts4bit_qlora); e4b, e4b-qlora, experts4bit, expertsnbit and experts-mxfp4 are lookup aliases. Two repositories: this one owns loading, quantisation orchestration, adapters, training, residency and serving; the kernels it calls through the [fast] extra live in grouped-nf4-gemm. Environment: Linux, a CUDA GPU, torch ≥ 2.2 and bitsandbytes ≥ 0.43 (Python 3.11 is what CI tests; the kernels need Triton on an sm_80+ GPU). The material limitation: on a model that already fits in bf16, 4-bit here is a memory trade, not a speed-up, and on the measured comparator it cost energy (e4b.train.energy-honest.scoped-a2000) — this is for models that do not fit. Machine-readable capabilities and evidence: docs/capabilities.json and docs/claims.json.

transformers v5 stores a MoE's experts as one fused 3-D parameter per layer. bitsandbytes' 4-bit walker only replaces nn.Linear, so it silently skips the experts — the overwhelming majority of the weights (bitsandbytes#1849). This package quantises exactly that fused stack (Experts4bit, the 4-bit face of ExpertsNbit: nf4 / fp4 / int8 / fp8 / bf16 / fp16 storage, with a test-pinned fidelity ordering), pairs it with a streaming loader and per-expert LoRA so you can fine-tune, and serves the result through a paged decode engine that is measured against each model's own attention.

Current position, one page: docs/STATUS.md. Every number, with its evidence and status: docs/claims.json.

Use this when

• load_in_4bit=True / BitsAndBytesConfig loads your MoE but the expert tensors (gate_up_proj, down_proj) stay bf16 and the model still OOMs — the experts are fused 3-D parameters, not nn.Linear.
• You need QLoRA or LoRA on the experts themselves, and PEFT or the bitsandbytes walker never sees them.
• The quantised experts fit in host RAM but not VRAM (stream per layer), or fit on NVMe but not host RAM (serve or train from an arena).
• You want to train a 30B-class MoE on a 12–24 GB consumer GPU (expert offload; e4b.offload.fits-30b-class), or serve one on an RTX 5090 — the only card the serving claims (e4b.serve.tp.*, e4b.serve.buildout.*) are measured on.
• You are choosing between the reference per-expert path, the batched path, the fused kernel path, host-streamed residency and the NVMe tier — docs/CHOOSING.md is the decision page.
• Your experts are NF4, native MXFP4 (gpt-oss, DeepSeek-V4), int4-b32 for serving, fp8 for the KV cache, or a mix across storage and residency tiers.

Do not use this when

• The model is dense (no experts): bitsandbytes' own 4-bit path already covers every nn.Linear.
• The model already fits in bf16 with headroom: 4-bit is a memory trade there, and on the measured comparator it was slower and used more energy (e4b.train.energy-honest.scoped-a2000 in the claims register — one card and one bitsandbytes development build, not a statement about every 4-bit path).
• You expect a general-purpose serving engine or a vLLM replacement: on the same box vLLM is ahead (e4b.serve.h2h.vllm.same-box); this is a measured 4-bit path for models that otherwise do not run at all.
• You need Windows, macOS, ROCm or a non-CUDA accelerator.
• The model family or expert layout is not in docs/ARCHITECTURE_SUPPORT.md — unsupported architectures fail fast with a named error; the accelerated paths fall back to the reference loop, so assert every enable_* count (a 0 looks identical to the per-expert loop).

Start here

Install

pip install experts4bit-qlora           # primitive + adapters (torch + bitsandbytes)
pip install "experts4bit-qlora[train]"  # + the streaming MoE trainer
pip install "experts4bit-qlora[fast]"   # + the fused grouped-GEMM path (grouped-nf4-gemm)

e4b, e4b-qlora, experts4bit, expertsnbit and experts-mxfp4 are lookup aliases that install this package; always install and cite experts4bit-qlora. Runs on stock bitsandbytes; every feature has a reference path. Building from source — pip install --no-build-isolation, or any build outside pip's isolated build environment — needs setuptools ≥ 77 for the PEP 639 license metadata in pyproject.toml; an ordinary pip install gets it automatically through build isolation.

Which door? Start from what does not fit

Reasoning and caveats for each: docs/CHOOSING.md. Assert the return value of every enable_*: 0 and "silently still on the per-expert loop" look identical from the caller's side.

Quickstart

import torch
from experts4bit_qlora import Experts4bit, ExpertsLoRA, load_moe_4bit_streaming, verify_moe_4bit

# A real fused-MoE checkpoint, quantised on the way to the GPU (never bf16-resident):
model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
)
verify_moe_4bit(model, strict=True)   # raises if any expert stack is still high precision

STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train      # QLoRA fine-tune
ADAPTER=./out/adapter_best.pt python -m experts4bit_qlora.infer                 # serve it

Do not load these models with stock from_pretrained(..., load_in_4bit=True): it quantises the nn.Linear layers, leaves the experts in bf16, and OOMs.

What is measured

Each row is an entry in docs/claims.json; the last column is its evidence status. measured means the receipt is in this repository; measured-private means the run happened but the receipt lives in a private audit tree and you cannot check it from here.

Three things to read beside that table, because they change what it means:

• A parity delta is read against a per-model noise floor, never against zero. Two arithmetically equivalent forwards of an MoE disagree, because rounding flips which experts the router picks; on gpt-oss 4.5% of layer-token choices flip and those tokens carry the whole disagreement. "Below the floor" means indistinguishable. docs/METHODOLOGY.md §13.1.
• 4-bit on a card that already fits the model was a 1.2–2.3× energy penalty on the measured comparator, not a saving: one OLMoE-dims expert projection on an RTX A2000, dequantize-then-linear and a bitsandbytes 0.50-dev fork build's matmul_4bit routing against native bf16 (e4b.train.energy-honest.scoped-a2000). It inverts when memory binds. Note, 2026-09-04: the earlier wording "NF4 is storage-only and the GEMM runs in bf16 either way" was a universal mechanism statement and is withdrawn as such — bitsandbytes ≥ 0.50.0 can run supported ordinary 2-D 4-bit inference cells on the packed weights directly, while routed grouped MoE execution and training's input gradient are separate contracts (docs/BITSANDBYTES.md). The measurement stands as its receipt made it.
• Ratios travel; absolutes do not. The 5090 class carries ~8.5% inter-box dispersion; the same config on two 4090s moved 8.6% in s/step. Quote the card, or quote a ratio.

What was retired

Claims this project published and then withdrew, each with the measurement that withdrew it, are listed in docs/STATUS.md and kept as retired entries in docs/claims.json so they stay findable. The most recent: the "+0.047 ppl fp8 KV cost" on Qwen3 (below the model's own floor), and the "+0.078 nats gpt-oss sinks/windows defect" (the chunked oracle was the drifting arm, not the serving path).

Scope

The primitives are model-agnostic. The streaming loader and trainer handle SwiGLU fused-MoE families stored per-expert or pre-fused: OLMoE, Qwen3-MoE / Qwen3.5-MoE, Gemma-4 (text tower), GraniteMoe, gpt-oss (MXFP4 experts with per-expert biases and a clamped GLU, dequantised bit-identically), and DeepSeek-V4 (Flash / Pro). Which families load, run and CUDA-graph-capture, with the evidence: docs/ARCHITECTURE_SUPPORT.md. Unsupported architectures fail fast with a clear error.

Known open: Gemma-4-26B-A4B's fp8 K cache wants finer groups on its 512-dim heads, and the family needs a parity instrument that survives its batch-shape variance (#359); the model fails to load on 2 of 5 rented hosts (#344); no shipped tool bakes the training arena from a bf16 checkpoint yet.

Docs

The package family

• experts4bit-qlora (this repo) owns everything around the expert GEMM: the fused-stack primitives and per-expert LoRA, the streaming loaders, offload, training, the paged serving engine, hot-expert residency.
• grouped-nf4-gemm owns the GEMM itself: one launch over 4-bit-packed expert stacks with in-register decode and fp32 accumulation, plus the fp8 paged decode attention and the decode glue kernels. [fast] is the seam.

The kernel makes one expert-stack matmul cheap; this package decides which bytes are where.

Provenance

Every number traces to a committed script and a named host, with receipts under bench/ and docs/ — or, where the receipt is private, the register says so. PROVENANCE.md is the OpenTimestamps-anchored record for the v0.2.0 convergence result; anchored documents are never edited in place (see docs/INDEX.md). Falsification work lives under audits/.

License

MIT (LICENSE). experts4bit_qlora/_vendor/experts.py is vendored from bitsandbytes (also MIT) pending upstream merge; its notice is in THIRD_PARTY_NOTICES.md.