Humming
Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference. It is developed by Ant Group and the vLLM community, and ships as the quantized GEMM backend in vLLM.
Read the launch blog: Announcing Humming: Fast, Flexible Kernels for Quantized Inference.
Key Features
- High Flexibility
- Supports inference for any weight type under 8-bit across FP16 / BF16 / FP8 / FP4 / INT8 / INT4 activations (provided the activation's dynamic range covers the weight type).
- Supports various quantization strategies (WNA16, W8A8, W4A8, W4A4, MXFP4/MXFP8, NVFP4, and more).
- Supports various scale types (BF16, FP16, E4M3, E5M2, and UE8M0).
- Supports both Dense GEMM and MoE GEMM (Indexed and Grouped layouts).
- Broad Hardware Support: every NVIDIA architecture from SM75 (Turing) onward, including Hopper, Blackwell (SM100/103/110/120/121), and Rubin, plus CUDA-compatible devices such as T-Head's Zhenwu 810E.
- High Performance: outperforms strong baselines (vLLM Marlin, FlashInfer, DeepGEMM) across many representative dense and MoE workloads. See Performance.
- Ultra-Lightweight
- Minimal dependencies: Requires only PyTorch and a CUDA compiler (NVCC or NVRTC).
- Compact footprint: the wheel is only about 500 KB; kernels are compiled on demand and cached, typically in a few seconds per layer.
Performance
Kernel microbenchmarks span M = 1 to 8192 with a cold L2 cache; MoE numbers sum the Gateup and Down GEMMs. Highlights:
| Workload | Device | Result |
|---|---|---|
| Dense W4A16 | Hopper / Blackwell | ~3–4× over cuBLAS at small M; up to ~2× vLLM Marlin on H200 and >3× on B300 at large M |
| MoE W4A16 (MXFP4 / MXINT4 / NVFP4) | H200 / B300 | ~2× vLLM Marlin on H200; up to ~5× on B300 (Grouped) |
| MoE W8A8 (FP8 block) | H20 / H200 | ~2.6× / ~2.5× DeepGEMM Contiguous at small–medium M |
| MoE W4A8 (MXFP4 × FP8) | H20 / H200 | ~4.2× / ~4.5× DeepGEMM W8A8, enabling fast MXFP4 on Hopper |
| MoE MX W8A8 / W4A8 / W4A4 | B300 | ~1.25× / 1.6× / 1.4× DeepGEMM at small–medium M |
| MoE W4A8 / W4A4 | SM120 / SM121 | Ahead of FlashInfer CUTLASS at most tested points |
| End-to-end DeepSeek-V4.1-Flash, 8×H20 | H20 | Humming W4A8 vs. Marlin W4A16: +14–43% throughput, −25–32% TTFT |
Full charts, setups, and software versions are in the launch blog.
Support Matrix
| Activation Type | Supported Devices | Supported Weight Types |
|---|---|---|
| FP16 (e5m10) | SM75+ | • Symmetric INT1-8 • INT1-8 with dynamic zero point • Arbitrary signed FP (kBits ≤ 8, kExp ≤ 5) |
| BF16 (e8m7) | SM80+ | • Symmetric INT1-8 • INT1-8 with dynamic zero point • Arbitrary signed FP (kBits ≤ 8) |
| FP8 (e4m3) | SM89+ | • Symmetric INT1-5 • INT1-4 with dynamic zero point • Arbitrary signed FP (kExp ≤ 4, kMan ≤ 3) |
| FP8 (e5m2) | SM89+ | • Symmetric INT1-4 • INT1-3 with dynamic zero point • Arbitrary signed FP (kExp ≤ 5, kMan ≤ 2) |
| FP4 (e2m1) | SM100+ | • Symmetric INT1-3 • INT1-2 with dynamic zero point • Arbitrary signed FP (kExp ≤ 2, kMan ≤ 1) |
| INT8 | SM75+ | • Symmetric INT1-8 • INT1-7 with dynamic zero point |
| INT4 | SM80+ | • Symmetric INT1-4 • INT1-3 with dynamic zero point |
On Blackwell (SM100/103/120/121), Humming also uses the hidden native float4e0m3 and float8e3m4 Tensor Core formats. SM103 (B300) has no INT8 UMMA, so INT8 there runs on the slower MMA path. The blog has a per-device chart of the high-performance activation types.
Getting Started
Installation
pip install humming-kernels
If your system has no CUDA compiler, pick the extra matching your PyTorch CUDA version:
pip install "humming-kernels[cu12]" # CUDA 12
pip install "humming-kernels[cu13]" # CUDA 13
On Linux, the extras install the CUDA compiler, headers and NVRTC at the toolkit version your PyTorch build pins, so they stay consistent with each other. There, cu13 requires PyTorch 2.11 or newer.
Or install from source:
pip install git+https://github.com/vllm-project/humming.git
Usage Example
import torch
from humming.layer import HummingLayer
layer = HummingLayer(
shape_n=8192,
shape_k=8192,
weight_config={"dtype": "int6"},
torch_dtype=torch.float16,
).cuda()
weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")
# Load unquantized weight and quantize to layer quantization format
layer.load_from_unquantized(weight)
# Transform weight to humming format and prepare default kernels
layer.transform()
# Run quantized GEMM (tuning_config is optional, auto-selected by default)
output = layer(inputs)
print("Quantized GEMM Output:")
print(output)
print("\nReference Output:")
print(inputs.matmul(weight.T))
See docs/config.md for the layer, compute, and tuning configuration reference.
Acknowledgement
This project is highly inspired by
- DeepGEMM
- Marlin Kernel and vLLM Marlin Kernel
- lmdeploy GEMM kernel
- FlashInfer
- CUTLASS
Metadata
Release files for humming-kernels 0.1.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| humming_kernels-0.1.20.tar.gz | 445.7 kB | With provenance | Details for humming_kernels-0.1.20.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| humming_kernels-0.1.20-cp311-abi3-win_arm64.whl | CPython 3.11 | abi3 | Windows ARM64 | Details for humming_kernels-0.1.20-cp311-abi3-win_arm64.whl |
| humming_kernels-0.1.20-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details for humming_kernels-0.1.20-cp311-abi3-win_amd64.whl |
| humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ x86-64 | Details for humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl |
| humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ ARM64 | Details for humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl |
Total release size: 2.8 MB
Release files / humming_kernels-0.1.20.tar.gz
| Download URL | humming_kernels-0.1.20.tar.gz |
|---|---|
| Size | 445.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
40c1fc66609bef617dc09ff8b9e32bde08db1c58f85a0e97a604f8995b5f838b
|
|
BLAKE2b-256 checksum How to use checksums |
62bc79507555b8a55e11b0b56fe83c91d7a2aaa9912bd4949ab45d0731abfdb1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / humming_kernels-0.1.20-cp311-abi3-win_arm64.whl
| Download URL | humming_kernels-0.1.20-cp311-abi3-win_arm64.whl |
|---|---|
| Size | 558.0 kB |
| Tags | CPython 3.11 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
e65b086862c578cd32b45d75bf1013aafb728be583bf62051cf875c2cb92901a
|
|
BLAKE2b-256 checksum How to use checksums |
ee288dec499ca5e17fddb9db7e56e9df2b6b279ad85b566bb6b65574bdc1aafe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / humming_kernels-0.1.20-cp311-abi3-win_amd64.whl
| Download URL | humming_kernels-0.1.20-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 567.2 kB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
3b75e4279ea5f0d4887fc344c98aca1bb69522d86ec3850a0cb77749524fa2c7
|
|
BLAKE2b-256 checksum How to use checksums |
be7a0e9f00cd824863f8f0c9a6680effc9afe7a042f1af93d43be6607e9fb588
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 616.4 kB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
d4747cae223f81bb8d18bee766a5f41e67484a65bd665534fdbc8dc7adabd7c1
|
|
BLAKE2b-256 checksum How to use checksums |
53b519eaeed76d26248b8fe9276d2f9a4346e250fd7d2cfb0784ae4929d7efcc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 601.9 kB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
1c7ecbeacb8f449136140a7fea68be06f2e5ce083e52de6b2e37e91158c3eee3
|
|
BLAKE2b-256 checksum How to use checksums |
07111d3961268098460ea32685ab687bdac1b6bbf681a488b82ef5a5098d872b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for humming-kernels 0.1.20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 5 files in this release have been uploaded with provenance metadata.
Attestation for humming_kernels-0.1.20.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.