experts4bit-qlora
Fine-tune large MoE models on a single GPU.
Quantize the experts. Train adapters. Move weights to RAM or SSD when GPU memory runs out.
On a 12 GB card: with OFFLOAD_EXPERTS=1, Qwen3-30B-A3B QLoRA-trains TC1's matched set (fp32 adapters, seq 2048,
r 16) at micro-batch 1 × accum 8 at 10.463 GB peak allocated on an RTX A2000 12 GB, while the rest of its packed experts
stay in pinned CPU RAM (e4b.train.frontier.qwen3.a2000-12gb.2026-10-02, one draw; micro-batch 2 ran out of memory on
that shared card). The command ·
how to read the number
Mixture-of-Experts (MoE) models keep many expert networks but activate only a few per token.
Their fused expert weights can escape ordinary load_in_4bit=True loading and stay in full precision.
This runtime loads those weights in 4-bit, trains them with QLoRA, and serves the model.
New here? Start with Loggetta.
pip install loggetta installs this runtime and its kernels. Use loggetta inspect to check your machine
and loggetta plan to plan supported training.
Use this package directly when you need control over loading, training, serving, or memory placement.
Limits. GPU execution needs Linux and NVIDIA CUDA; the fast kernels need an sm_80+ GPU. Support varies by model family and execution path. 4-bit saves memory but is not automatically faster than bf16, and vLLM was faster in the measured Qwen decode comparison. Check the fit.
Install · Results · Model support · Task guides · Hugging Face
Latest released package: experts4bit-qlora 0.56.1 · Current development status: docs/STATUS.md on main (this README describes main) · Released documentation for 0.56.1: docs/ · README.md · CHANGELOG.md
Install
pip install "experts4bit-qlora[train,fast]"
GPU execution needs Linux + NVIDIA CUDA. Fast kernels need an sm_80+ GPU, Triton ≥ 3.4; the streaming loader needs transformers ≥ 5.0. The base runtime needs torch ≥ 2.2; the fast extra applies the kernel package’s newer PyTorch floor. CI tests Python 3.11. Start with a CUDA-compatible PyTorch installation for your GPU.
For a smaller install, the base package provides quantization and adapters; [train] adds the trainer;
[fast] adds grouped-nf4-gemm.
Exact dependency floors live in pyproject.toml.
Load a model and check it
This downloads Qwen3-30B-A3B and keeps its frozen experts in host RAM. Allow enough RAM and disk space for the checkpoint.
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit
model, config = load_moe_4bit_streaming(
"Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16,
r=8, alpha=16, quant_type="nf4", offload=True,
)
verify_moe_4bit(model, strict=True) # raises if an expert stack stayed high precision
To train the built-in Alpaca example:
MODEL=Qwen/Qwen3-30B-A3B OFFLOAD_EXPERTS=1 STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train
For your own dataset and reusable native adapters, see Loggetta's training guide.
Choose the path you need
| Task | Start here |
|---|---|
| Plan before loading weights | Loggetta |
| Train fused MoE experts with QLoRA | Training guide |
| Speed up training | enable_fast_train(model, dgrad=True); assert its return count is positive |
| Fit experts in RAM or NVMe storage | Offload guide |
| Fit the rest of the model | enable_dense_offload — freeze base weights before adding adapters |
| Serve on a consumer GPU | Serving guide |
| Work with native MXFP4 checkpoints | MXFP4 guide |
What is measured
These are specific workloads, with public results and controls. The table is checked against claims.json in CI. Measured means a public run record; the full register also labels private, superseded, and retired results.
| Workload and evidence | Result | Status |
|---|---|---|
Qwen3-30B-A3B QLoRA · RTX 5090 · TC1's field recipe, both frameworks on one stack, e4b at every default (e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio) |
Unsloth spends 1.92× e4b's GPU time per step; wall-clock 2.80× on an AMD EPYC 7713. Comparable held-out loss. Peak 26.16 vs 24.27 GB (Unsloth lower). | measured |
Packed 4,096-token Qwen3 training · RTX 5090 · e4b at its defaults (e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio) |
Unsloth spends 1.160× e4b's GPU time per step; the wall-clock ratio varies with the host (below). | measured |
Expert-offload training on a 12 GB card (e4b.train.frontier.qwen3.a2000-12gb.2026-10-02) |
Qwen3-30B-A3B, TC1's matched set (fp32 adapters) at micro-batch 1 × accum 8: 10.463 GB peak allocated on an RTX A2000 12 GB, held-out 1.9533 → 0.8483 at N=20. An older, lighter reading gave 7.16 GB, and 8.47 GB for Gemma-4-26B-A4B (e4b.offload.fits-30b-class); it was not reproduced (how to read these). |
measured |
The speed rows profile both frameworks on one stack (torch 2.12.1). The GPU-time ratio is the host-independent figure; the
wall-clock ratio grows on a slower host, because Unsloth runs 14× e4b's CPU ops per step at the field recipe. On packed rows
the wall ratio read 1.472 (e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio) and 1.773
(e4b.train.memory.packed-4k-position-defaults.5090.2026-10-07) on two hosts. The earlier field-recipe readings, 2.352 and 2.468, were wall
ratios from before the reentrant checkpoint.
The offload row uses a different training setup.
Qwen comparison · Packed 4K result · All current results
How the two Unsloth rows were measured
- Hardware: one rented RTX 5090 per reading (a Vast.ai verified host). The field-recipe row ran on an AMD EPYC 7713 host; the packed row on an AMD engineering-sample CPU with a 30.7-CPU quota.
- Software: torch 2.12.1+cu130 for both frameworks; Unsloth 2026.9.14; e4b and grouped-nf4-gemm at the commits in each run record.
- Method: matched fp32 adapters and initialization, two draws per arm, three profiled steps before the timed window.
Validity and scoring by
bench/tc1/tc1_reduce.py, under the pre-registration. - Rerun: the TC1 harness on one RTX 5090, token
qwen3pos69(field recipe) orqwen3pos68(packed rows). - Run records (versions, host and every arm): field recipe · packed rows
New in 0.56.1
- Sixteen requests decode 1.4654× as fast on Qwen3-30B-A3B, one request unchanged, every token identical. K25, the
batched NF4 decode GEMM, now runs at tile plan
c1onqwen3_moewhere its decode resolved to K39'sprmt(grouped-nf4-gemm 0.50.0 or later, on NVIDIA). Elsewhere it stays ontoday, recorded ininfo["k25_plan"]. The plan changes tiling and staging only, so the values are bit-identical totoday's. Measured on one RTX 5090 against thetodayplan, both under K39's decode: the smallest of four ABBA pairs.E4B_K25_PLAN=todayopts out. (P161,e4b.serve.p161.k25-plan-c1.qwen3.5090.2026-10-20)
New in 0.56.0
Serving defaults on qwen3_moe. Each was measured alone on Qwen3-30B-A3B on one RTX 5090 (170 SMs); their combined
effect has not been read. Two need a newer grouped-nf4-gemm than [fast] installs (0.43.0 or later): K25's prmt decode
needs 0.50.0 and the int8 head needs 0.48.0. Below those each resolves off, recorded, and nothing breaks. To get them,
run pip install -U grouped-nf4-gemm.
- Sixteen requests decode 1.0925× as fast, one request unchanged, every token identical. K25, the batched NF4 decode
GEMM, now decodes its codebook with grouped-nf4-gemm K39's byte-permute decode (
lut="prmt"), bit-identical to the select tree it replaces. It needs grouped-nf4-gemm 0.50.0 or later and an NVIDIA device; otherwise it resolves to the tree, recorded ininfo["k25_lut"].E4B_K25_LUT=treeopts out. (P158,e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20) - One request decodes 1.052× as fast from the lm_head on its per-row int8 grid (grouped-nf4-gemm K36; at
Qwen3-30B-A3B's 151936 x 2048 head only; needs grouped-nf4-gemm 0.48.0 or later, else it resolves off). Outputs
change: the logits differ from the bf16 head's, inside the quality gates read on Qwen3-30B-A3B @ad44e777. Sixteen
requests read 0.992, a cost not yet attributed.
E4B_INT8_HEAD_BW=0restores the bf16 head. (P153,e4b.serve.p153.int8-head-bw.qwen3.5090.2026-10-19) - One request decodes 1.044× as fast with the paged FP8 attention decode capped at 16 splits (buckets 1, 2 and 4
move; bitwise-identical to the library's plan at contexts up to 1024). Sixteen requests are unchanged.
E4B_ATTN_SPLIT_CAP=0opts out. (P143,e4b.serve.p143.attn-split-cap.qwen3.5090.2026-10-19)
Training, opt-in. Each was measured on the card named, against the baseline its bullet names.
- Layer-major training under expert offload steps 2.365× as fast on Qwen3-30B-A3B on one RTX 4090.
layer_major_step(experts4bit_qlora.engines.layer_major) runs a step's micro-batches layer by layer, so each offloaded layer is staged once per step instead of once per micro-batch. It replicates at 2.311× on an RTX 5090 and holds on Mixtral-8x7B at 2.118× on an RTX 4090. Opt-in. (P137,e4b.train.p137.layer-major-offload.qwen3.4090.2026-10-18; P140,e4b.train.p140.layer-major-offload.qwen3.5090.2026-10-18; P144,e4b.train.p144.layer-major-offload.mixtral.4090.2026-10-18) - The CLI trainer can take the fused training path. In 0.55.0,
python -m experts4bit_qlora.trainalways ran the reference loop.FAST_TRAIN=1now puts it on the fused path, andLAYER_MAJOR=1implies it. At the CLI's own default recipe on Qwen3-30B-A3B withOFFLOAD_EXPERTS=1,LAYER_MAJOR=1steps 1.455× as fast asFAST_TRAIN=1alone on one RTX 4090. An explicitFAST_TRAIN=0keeps the reference loop. (P146,e4b.train.p146.layer-major-cli.qwen3.4090.2026-10-18)
Mixtral serving.
- Mixtral-8x7B decodes with its B=1 fold stack by default. The T1 glue, its second round and the router epilogue now
resolve to
autoon Mixtral; fused q/k/v does not engage on this family. A one-request step takes 0.9393 of its time without them (about 20.2-20.5 ms to 19.0 ms) on one RTX 5090. Opt out per knob:E4B_FUSE_T1_GLUE=0,E4B_FUSE_T1_GLUE_R2=0,E4B_FUSE_ROUTER_EPI=0. (P135,e4b.serve.p135.fold-speed.mixtral.5090.2026-10-18) E4B_SERVE_ATTN_INT8=1: int8 weight-only attention projections on Mixtral, opt-in. q/k/v/o are held in int8 with a per-row fp32 scale, and decode calls of 1-16 rows run grouped-nf4-gemm's K37 kernels (from 0.49.0). On one RTX 5090 the served read decodes one request 1.161x as fast and sixteen 1.018x, with decode quality inside its registered bound. It stays opt-in until a routing-matched read: the default-flip PR is held. (P149,e4b.serve.p149.int8-attn.mixtral.5090.2026-10-20)
Scale.
- One full-depth QLoRA optimizer step on Qwen3-235B-A22B on one GPU. All 94 decoder layers and every routed expert,
two optimizer steps plus a fresh-process repeat of step 0, on one RTX PRO 6000 Blackwell Workstation Edition.
(Q235, run
q235-pro6000-8, MILESTONE PASS)
Check the fit
- Models: use the architecture support matrix. Support and test coverage vary by family and execution path.
- Memory: CPU/NVMe offload trades transfer time and host storage for GPU memory. Measure your workload.
- Speed: 4-bit saves weight memory; it is not automatically faster or more energy-efficient than bf16 when both fit.
- Serving: vLLM was faster in the measured Qwen decode comparison. This runtime focuses on fitting and controlling low-bit MoEs; see the serving results.
- Platforms: CUDA execution is the tested path. ROCm, macOS, Windows and other non-CUDA GPU execution are outside this package's scope.
Documentation and machine-readable data
Task guides · API and capability map · Claims and evidence · Package compatibility · Documentation index · llms.txt · Contributing
License
MIT. The vendored bitsandbytes code is also MIT; see third-party notices. For integration or research work: jordananderson.work.
1 maintainer
Metadata
1 maintainer
Release files for experts4bit-qlora 0.56.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| experts4bit_qlora-0.56.1.tar.gz | 2.9 MB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| experts4bit_qlora-0.56.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.7 MB
Release files / experts4bit_qlora-0.56.1.tar.gz
| Download URL | experts4bit_qlora-0.56.1.tar.gz |
|---|---|
| Size | 2.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f4e124ff3f5fe485559748a6bb9499e91b9db2059b6eea6972904e031b020613
|
|
BLAKE2b-256 checksum How to use checksums |
f0a225697a8c7c9fc2b4b5552768be0617a9042862cbe95ed17b09e936d7733a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logRelease files / experts4bit_qlora-0.56.1-py3-none-any.whl
| Download URL | experts4bit_qlora-0.56.1-py3-none-any.whl |
|---|---|
| Size | 742.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
56393015ef88508eaca9c369f52c4de3337105929ded4212a8974b7e4cee5476
|
|
BLAKE2b-256 checksum How to use checksums |
42504c75493b0e6328fd1eb6440ff33147eb5ad3775d6686527310b997fd530c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logSecurity for experts4bit-qlora 0.56.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for experts4bit_qlora-0.56.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
