experts4bit-qlora
Fine-tune large MoE models on a single GPU.
Quantize the experts. Train adapters. Move weights to RAM or SSD when GPU memory runs out.
Mixture-of-Experts (MoE) models keep many expert networks but activate only a few per token.
Their fused expert weights can escape ordinary load_in_4bit=True loading and stay in full precision.
This runtime loads those weights in 4-bit, trains them with QLoRA, and serves the model.
New here? Start with Loggetta.
pip install loggetta installs this runtime and its kernels. Use loggetta inspect to check your machine
and loggetta plan to plan supported training.
Use this package directly when you need control over loading, training, serving, or memory placement.
Install · Results · Model support · Task guides
Latest released package: experts4bit-qlora 0.49.0 · Current development status: docs/STATUS.md on main (this README describes main) · Released documentation for 0.49.0: docs/ · README.md · CHANGELOG.md
Install
pip install "experts4bit-qlora[train,fast]"
GPU execution needs Linux + NVIDIA CUDA. Fast kernels need an sm_80+ GPU, Triton ≥ 3.4; the streaming loader needs transformers ≥ 5.0. The base runtime needs torch ≥ 2.2; the fast extra applies the kernel package’s newer PyTorch floor. CI tests Python 3.11. Start with a CUDA-compatible PyTorch installation for your GPU.
For a smaller install, the base package provides quantization and adapters; [train] adds the trainer;
[fast] adds grouped-nf4-gemm.
Exact dependency floors live in pyproject.toml.
Load a model and check it
This downloads Qwen3-30B-A3B and keeps its frozen experts in host RAM. Allow enough RAM and disk space for the checkpoint.
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit
model, config = load_moe_4bit_streaming(
"Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16,
r=8, alpha=16, quant_type="nf4", offload=True,
)
verify_moe_4bit(model, strict=True) # raises if an expert stack stayed high precision
To train the built-in Alpaca example:
MODEL=Qwen/Qwen3-30B-A3B OFFLOAD_EXPERTS=1 STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train
For your own dataset and reusable native adapters, see Loggetta's training guide.
Choose the path you need
| Task | Start here |
|---|---|
| Plan before loading weights | Loggetta |
| Train fused MoE experts with QLoRA | Training guide |
| Speed up training | enable_fast_train(model, dgrad=True); assert its return count is positive |
| Fit experts in RAM or NVMe storage | Offload guide |
| Fit the rest of the model | enable_dense_offload — freeze base weights before adding adapters |
| Serve on a consumer GPU | Serving guide |
| Work with native MXFP4 checkpoints | MXFP4 guide |
What is measured
These are specific workloads, with public results and controls. The table is checked against claims.json in CI. Measured means a public run record; the full register also labels private, superseded, and retired results.
| Workload and evidence | Result | Status |
|---|---|---|
Qwen3-30B-A3B QLoRA · RTX 5090 · matched work, same software stack (e4b.train.h2h.unsloth.qwen3.5090.2026-10-05.same-stack) |
2.352× training speed vs Unsloth; 3.494 vs 8.218 s/step. Comparable held-out loss. Unsloth used 3.22 GB less peak VRAM. | measured |
Same comparison on a second RTX 5090 host (e4b.train.h2h.unsloth.qwen3.5090.2026-10-05.same-stack-host2) |
2.468× training speed vs Unsloth. | measured |
Packed 4,096-token Qwen3 training · RTX 5090 · defaults as of 2026-10-06 (e4b.train.h2h.unsloth.qwen3.5090.2026-10-06.packed-4k-defaults) |
1.453× training speed vs Unsloth; 28.23 vs 24.86 GB peak. Held-out loss 0.9544 vs 0.9543. | measured |
Expert-offload training on a 12 GB card (e4b.offload.fits-30b-class) |
Qwen3-30B-A3B: 7.16 GB peak; Gemma-4-26B-A4B: 8.47 GB. | measured |
The speed rows use torch 2.12.1+cu130 and transformers 5.5.0. The packed result uses
pre-release 0.49.0 code and grouped-nf4-gemm 0.42.0. Its PyTorch 2.8 run was slower, and bucketed padding is
not the cause: under 2.8 it steps 0.983× the single block with a 4.24 GB lower peak (e4b.train.pad-buckets.torch28.qwen3.5090.2026-10-06).
Four training defaults changed after that run. At the new defaults e4b's packed-row training peaks at 26.58 GB, Unsloth's at
24.86 (e4b.train.memory.packed-4k-train-census-defaults.5090.2026-10-07). Speed has not been re-measured at them.
The offload row uses a different training setup.
Qwen comparison · Packed 4K result · All current results
New in 0.49.0
- Faster, leaner training by default under
enable_fast_train: the reentrant checkpoint, the compressed expert absmax, the chunked held-out and language-model losses, and the combine over row chunks. Each has an off switch (CHANGELOG). - Serving:
serve_pageddoes its KV bookkeeping in bulk, which doubled one RTX 5090's capacity (2 → 4 req/s). - Compatibility: works with transformers 5.19.0 again.
- With grouped-nf4-gemm 0.43.0 (
pip install -U grouped-nf4-gemm): single-request decode 1.24× as fast on GPUs with 160+ SMs, and 0.65 GB less training memory on packed rows.
Check the fit
- Models: use the architecture support matrix. Support and test coverage vary by family and execution path.
- Memory: CPU/NVMe offload trades transfer time and host storage for GPU memory. Measure your workload.
- Speed: 4-bit saves weight memory; it is not automatically faster or more energy-efficient than bf16 when both fit.
- Serving: vLLM was faster in the measured Qwen decode comparison. This runtime focuses on fitting and controlling low-bit MoEs; see the serving results.
- Platforms: CUDA execution is the tested path. ROCm, macOS, Windows and other non-CUDA GPU execution are outside this package's scope.
Documentation and machine-readable data
Task guides · API and capability map · Claims and evidence · Package compatibility · Documentation index · llms.txt · Contributing
License
MIT. The vendored bitsandbytes code is also MIT; see third-party notices. For integration or research work: jordananderson.work.
1 maintainer
Metadata
1 maintainer
Release files for experts4bit-qlora 0.49.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| experts4bit_qlora-0.49.0.tar.gz | 1.9 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| experts4bit_qlora-0.49.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.5 MB
Release files / experts4bit_qlora-0.49.0.tar.gz
| Download URL | experts4bit_qlora-0.49.0.tar.gz |
|---|---|
| Size | 1.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
01fca62e8eb80ee3750fb68fcfcf503fa1b0ebb06e5dbe1960fdab8935224317
|
|
BLAKE2b-256 checksum How to use checksums |
f9a6fae57acf29ec6d320fc8e6caf62a0385d6df0befd210269a2e99b5217480
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.
Transparency logRelease files / experts4bit_qlora-0.49.0-py3-none-any.whl
| Download URL | experts4bit_qlora-0.49.0-py3-none-any.whl |
|---|---|
| Size | 642.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fe5ba350ddafe7cea151c59b766102371f77e0619ab7b87e1cc621a5ed86adb8
|
|
BLAKE2b-256 checksum How to use checksums |
ba6485e5a429afd2c81c651e180893959a7550d88381b156edf7a5cde07cc1a5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.
Transparency log
