experts4bit-qlora

Latest released package: experts4bit-qlora 0.37.2 · Current development status: docs/STATUS.md on main (this README describes main) · Released documentation for 0.37.2: docs/ · README.md · CHANGELOG.md

Train and serve fused Mixture-of-Experts models in 4-bit on hardware that cannot hold them in bf16.

The problem in one line: load_in_4bit=True leaves a fused MoE's expert weights in bf16, so the model still OOMs; this package quantises exactly those experts, fine-tunes them with QLoRA, keeps them in host RAM or on NVMe when they do not fit, and serves them on one consumer NVIDIA GPU. Canonical package: experts4bit-qlora on PyPI (import experts4bit_qlora); e4b, e4b-qlora, experts4bit, expertsnbit and experts-mxfp4 are lookup aliases. Two repositories: this one owns loading, quantisation orchestration, adapters, training, residency and serving; the kernels it calls through the [fast] extra live in grouped-nf4-gemm. Environment: Linux, a CUDA GPU, torch ≥ 2.2 and bitsandbytes ≥ 0.43 (the floors are pyproject.toml's; Python 3.11 is what CI tests; the kernels need Triton on an sm_80+ GPU). The material limitation: on a model that already fits in bf16, 4-bit here is a memory trade, not a speed-up, and on the measured comparator it cost energy (e4b.train.energy-honest.scoped-a2000) — this is for models that do not fit. Machine-readable capabilities and evidence: docs/capabilities.json and docs/claims.json.

transformers v5 stores a MoE's experts as one fused 3-D parameter per layer. bitsandbytes' 4-bit walker only replaces nn.Linear, so it silently skips the experts — the overwhelming majority of the weights (bitsandbytes#1849). This package quantises exactly that fused stack (Experts4bit, the 4-bit face of ExpertsNbit: nf4 / fp4 / int8 / fp8 / bf16 / fp16 storage, with a test-pinned fidelity ordering), pairs it with a streaming loader and per-expert LoRA so you can fine-tune, and serves the result through a paged decode engine that is measured against each model's own attention.

Current position, one page: docs/STATUS.md. Every number, with its evidence and status: docs/claims.json. This README describes main: every link is to main, every number is the current value of the claim it names, and the documentation as released with a version is reached from the release block at the top.

Use this when

• load_in_4bit=True / BitsAndBytesConfig loads your MoE but the expert tensors (gate_up_proj, down_proj) stay bf16 and the model still OOMs — the experts are fused 3-D parameters, not nn.Linear.
• You need QLoRA or LoRA on the experts themselves, and PEFT or the bitsandbytes walker never sees them.
• The quantised experts fit in host RAM but not VRAM (stream per layer), or fit on NVMe but not host RAM (serve or train from an arena).
• You want to train a 30B-class MoE on a 12–24 GB consumer GPU (expert offload; e4b.offload.fits-30b-class), or serve one on an RTX 5090 — the only card the serving claims (e4b.serve.census.bo7.*, the position; e4b.serve.buildout.* behind it) are measured on.
• You are choosing between the reference per-expert path, the batched path, the fused kernel path, host-streamed residency and the NVMe tier — docs/CHOOSING.md is the decision page.
• Your experts are NF4, native MXFP4 (gpt-oss, DeepSeek-V4), int4-b32 for serving, fp8 for the KV cache, or a mix across storage and residency tiers.

Do not use this when

• The model is dense (no experts): bitsandbytes' own 4-bit path already covers every nn.Linear.
• The model already fits in bf16 with headroom: 4-bit is a memory trade there, and on the measured comparator it was slower and used more energy (e4b.train.energy-honest.scoped-a2000 in the claims register — one card and one bitsandbytes development build, not a statement about every 4-bit path).
• You expect a general-purpose serving engine or a vLLM replacement: on the same box, with identical prompt ids, vLLM 0.30.0 decodes 1.087× faster than this package's current int4 stack at B=1 and 1.396× at B=16 (e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b1.5090.2026-09-22, e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b16.5090.2026-09-22; bounded to graph decode at B=1 and B=16 on one RTX 5090 box with one prompt set — never a general position; vLLM's number includes its serving loop and this package's does not, so the engine advantage is understated; quality is quoted, never equated; footprint and prefill/TTFT were not compared); this is a measured 4-bit path for models that otherwise do not run at all.
• You need Windows, macOS, ROCm or a non-CUDA accelerator.
• The model family or expert layout is not in docs/ARCHITECTURE_SUPPORT.md — unsupported architectures fail fast with a named error; the accelerated paths fall back to the reference loop, so assert every enable_* count (a 0 looks identical to the per-expert loop).

Start here

Install

pip install experts4bit-qlora           # primitive + adapters (torch + bitsandbytes)
pip install "experts4bit-qlora[train]"  # + the streaming MoE trainer
pip install "experts4bit-qlora[fast]"   # + the fused grouped-GEMM path (grouped-nf4-gemm)

e4b, e4b-qlora, experts4bit, expertsnbit and experts-mxfp4 are lookup aliases that install this package; always install and cite experts4bit-qlora. Runs on stock bitsandbytes; every feature has a reference path. Building from source — pip install --no-build-isolation, or any build outside pip's isolated build environment — needs setuptools ≥ 77 for the PEP 639 license metadata in pyproject.toml; an ordinary pip install gets it automatically through build isolation. The [fast] extra's floor on grouped-nf4-gemm is pyproject.toml's and is not repeated here; which version of this package needs which kernel release, and why, is the compatibility record in docs/system-manifest.json, validated in CI against pyproject.toml.

Which door? Start from what does not fit

Reasoning and caveats for each: docs/CHOOSING.md. Assert the return value of every enable_*: 0 and "silently still on the per-expert loop" look identical from the caller's side.

Quickstart

import torch
from experts4bit_qlora import Experts4bit, ExpertsLoRA, load_moe_4bit_streaming, verify_moe_4bit

# A real fused-MoE checkpoint, quantised on the way to the GPU (never bf16-resident):
model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
)
verify_moe_4bit(model, strict=True)   # raises if any expert stack is still high precision

STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train      # QLoRA fine-tune
ADAPTER=./out/adapter_best.pt python -m experts4bit_qlora.infer                 # serve it

Do not load these models with stock from_pretrained(..., load_in_4bit=True): it quantises the nn.Linear layers, leaves the experts in bf16, and OOMs.

What is measured

Each row names its entries in docs/claims.json, which carry the value, the conditions and the receipt path; the last column is the status there. measured means the receipt is in this repository; measured-private means the run happened but the receipt lives in a private audit tree and you cannot check it from here. Every number in the result column is the named claim's current value: scripts/check_readme_claims.py holds this table to the register in CI and fails on drift, on a superseded or retired id, and on a private receipt presented as public — a number that moves in the register moves here or the build goes red.

Three things to read beside that table, because they change what it means:

• A parity delta is read against a per-model noise floor, never against zero. Two arithmetically equivalent forwards of an MoE disagree, because rounding flips which experts the router picks; on gpt-oss 4.5% of layer-token choices flip and those tokens carry the whole disagreement. "Below the floor" means indistinguishable. docs/METHODOLOGY.md §13.1.
• 4-bit on a card that already fits the model was a 1.2–2.3× energy penalty on the measured comparator, not a saving: one OLMoE-dims expert projection on an RTX A2000, dequantize-then-linear and a bitsandbytes 0.50-dev fork build's matmul_4bit routing against native bf16 (e4b.train.energy-honest.scoped-a2000). It inverts when memory binds. Note, 2026-09-04: the earlier wording "NF4 is storage-only and the GEMM runs in bf16 either way" was a universal mechanism statement and is withdrawn as such — bitsandbytes ≥ 0.50.0 can run supported ordinary 2-D 4-bit inference cells on the packed weights directly, while routed grouped MoE execution and training's input gradient are separate contracts (docs/BITSANDBYTES.md). The measurement stands as its receipt made it.
• A head-to-head is one workload on one box. The Unsloth row above is ≈86 tokens per step at batch 1 on a resident 30B MoE; its 200-step curve favours Unsloth and is quoted beside the 60-step position wherever that position is quoted (bench/h2h-20260905/p38/, the pre-registration and every amendment in the bundle). The vLLM row (bench/h2h-20260905/p37/) quotes vLLM against this package's slowest, licence-free configuration because that is the only ratio the lane could quote: the licensed arms were void on that box (a pack fingerprint that did not reproduce, and the registered gate run on that pack failed its second text), so the number a reader wants — vLLM against the licensed stack — does not exist on that lane; what exists is an open reproducibility item on the streamed calibration (docs/STATUS.md). The 2026-09-03 comparison (×1.47 / ×1.55) is superseded and stays as history.
• Ratios travel; absolutes do not. The 5090 class carries ~8.5% inter-box dispersion; the same config on two 4090s moved 8.6% in s/step. Quote the card, or quote a ratio.

What was retired

Claims this project published and then withdrew, each with the measurement that withdrew it, are listed in docs/STATUS.md and kept as retired entries in docs/claims.json so they stay findable. The most recent: the "+0.047 ppl fp8 KV cost" on Qwen3 (below the model's own floor), and the "+0.078 nats gpt-oss sinks/windows defect" (the chunked oracle was the drifting arm, not the serving path).

Scope

The primitives are model-agnostic. The streaming loader admits 14 fused-MoE families (SUPPORTED_ARCHITECTURES), stored per-expert or pre-fused: OLMoE, Qwen3-MoE / Qwen3.5-MoE, Gemma-4 (text tower), gpt-oss (MXFP4 experts with per-expert biases and a clamped GLU, dequantised bit-identically), GraniteMoe with its shared-expert and Mamba-hybrid variants (granite-4.0-h), Nemotron-H (non-gated ReLU² experts), LFM2-MoE, Jamba, Kimi K3 and DeepSeek-V4 (Flash / Pro). Ten of the 14 have loaded a real published checkpoint; which ones, which load, run and CUDA-graph-capture, and the five families that have a convention but no loader path yet (each with the blocker it is waiting on) are in docs/ARCHITECTURE_SUPPORT.md. Unsupported architectures fail fast with a clear error. Which of those families have a training receipt on real weights under the shipped code, per path (quantize, reference, fused, batched, NVMe, native MXFP4), and which enablers refuse: the tp1 section of the same document and training_support in docs/capabilities.json — the capability's model_families (olmoe, qwen3_moe, gemma4_text, mixtral, granitemoe) is exactly the families whose fused path passes with a receipt here; gpt-oss is refused, its experts trainable only through the kernel package's experimental MXFP4 route.

Known open: Gemma-4-26B-A4B's fp8 K cache wants finer groups on its 512-dim heads, and the family needs a parity instrument that survives its batch-shape variance (#359); the model fails to load on 2 of 5 rented hosts (#344); no shipped tool bakes the training arena from a bf16 checkpoint yet.

Docs

Every link is to main, the current position; the documentation as it was released with a version is linked from the release block at the top.

The package family

• experts4bit-qlora (this repo) owns everything around the expert GEMM: the fused-stack primitives and per-expert LoRA, the streaming loaders, offload, training, the paged serving engine, hot-expert residency.
• grouped-nf4-gemm owns the GEMM itself: one launch over 4-bit-packed expert stacks with in-register decode and fp32 accumulation, plus the fp8 paged decode attention and the decode glue kernels. [fast] is the seam.

The kernel makes one expert-stack matmul cheap; this package decides which bytes are where.

Provenance

Every number traces to a committed script and a named host, with receipts under bench/ and docs/ — or, where the receipt is private, the register says so. PROVENANCE.md is the OpenTimestamps-anchored record for the v0.2.0 convergence result; anchored documents are never edited in place (see docs/INDEX.md). Falsification work lives under audits/.

License

MIT (LICENSE). experts4bit_qlora/_vendor/experts.py is vendored from bitsandbytes (also MIT) pending upstream merge; its notice is in THIRD_PARTY_NOTICES.md.