Skip to main content

Humming

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference. It is developed by Ant Group and the vLLM community, and ships as the quantized GEMM backend in vLLM.

Read the launch blog: Announcing Humming: Fast, Flexible Kernels for Quantized Inference.

Key Features

  • High Flexibility
    • Supports inference for any weight type under 8-bit across FP16 / BF16 / FP8 / FP4 / INT8 / INT4 activations (provided the activation's dynamic range covers the weight type).
    • Supports various quantization strategies (WNA16, W8A8, W4A8, W4A4, MXFP4/MXFP8, NVFP4, and more).
    • Supports various scale types (BF16, FP16, E4M3, E5M2, and UE8M0).
    • Supports both Dense GEMM and MoE GEMM (Indexed and Grouped layouts).
  • Broad Hardware Support: every NVIDIA architecture from SM75 (Turing) onward, including Hopper, Blackwell (SM100/103/110/120/121), and Rubin, plus CUDA-compatible devices such as T-Head's Zhenwu 810E.
  • High Performance: outperforms strong baselines (vLLM Marlin, FlashInfer, DeepGEMM) across many representative dense and MoE workloads. See Performance.
  • Ultra-Lightweight
    • Minimal dependencies: Requires only PyTorch and a CUDA compiler (NVCC or NVRTC).
    • Compact footprint: the wheel is only about 500 KB; kernels are compiled on demand and cached, typically in a few seconds per layer.

Performance

Kernel microbenchmarks span M = 1 to 8192 with a cold L2 cache; MoE numbers sum the Gateup and Down GEMMs. Highlights:

Workload Device Result
Dense W4A16 Hopper / Blackwell ~3–4× over cuBLAS at small M; up to ~2× vLLM Marlin on H200 and >3× on B300 at large M
MoE W4A16 (MXFP4 / MXINT4 / NVFP4) H200 / B300 ~2× vLLM Marlin on H200; up to ~5× on B300 (Grouped)
MoE W8A8 (FP8 block) H20 / H200 ~2.6× / ~2.5× DeepGEMM Contiguous at small–medium M
MoE W4A8 (MXFP4 × FP8) H20 / H200 ~4.2× / ~4.5× DeepGEMM W8A8, enabling fast MXFP4 on Hopper
MoE MX W8A8 / W4A8 / W4A4 B300 ~1.25× / 1.6× / 1.4× DeepGEMM at small–medium M
MoE W4A8 / W4A4 SM120 / SM121 Ahead of FlashInfer CUTLASS at most tested points
End-to-end DeepSeek-V4.1-Flash, 8×H20 H20 Humming W4A8 vs. Marlin W4A16: +14–43% throughput, −25–32% TTFT

Full charts, setups, and software versions are in the launch blog.

Support Matrix

Activation Type Supported Devices Supported Weight Types
FP16 (e5m10) SM75+ • Symmetric INT1-8
• INT1-8 with dynamic zero point
• Arbitrary signed FP (kBits ≤ 8, kExp ≤ 5)
BF16 (e8m7) SM80+ • Symmetric INT1-8
• INT1-8 with dynamic zero point
• Arbitrary signed FP (kBits ≤ 8)
FP8 (e4m3) SM89+ • Symmetric INT1-5
• INT1-4 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 4, kMan ≤ 3)
FP8 (e5m2) SM89+ • Symmetric INT1-4
• INT1-3 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 5, kMan ≤ 2)
FP4 (e2m1) SM100+ • Symmetric INT1-3
• INT1-2 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 2, kMan ≤ 1)
INT8 SM75+ • Symmetric INT1-8
• INT1-7 with dynamic zero point
INT4 SM80+ • Symmetric INT1-4
• INT1-3 with dynamic zero point

On Blackwell (SM100/103/120/121), Humming also uses the hidden native float4e0m3 and float8e3m4 Tensor Core formats. SM103 (B300) has no INT8 UMMA, so INT8 there runs on the slower MMA path. The blog has a per-device chart of the high-performance activation types.

Getting Started

Installation

pip install humming-kernels

If your system has no CUDA compiler, pick the extra matching your PyTorch CUDA version:

pip install "humming-kernels[cu12]"  # CUDA 12
pip install "humming-kernels[cu13]"  # CUDA 13

On Linux, the extras install the CUDA compiler, headers and NVRTC at the toolkit version your PyTorch build pins, so they stay consistent with each other. There, cu13 requires PyTorch 2.11 or newer.

Or install from source:

pip install git+https://github.com/vllm-project/humming.git

Usage Example

import torch
from humming.layer import HummingLayer

layer = HummingLayer(
    shape_n=8192,
    shape_k=8192,
    weight_config={"dtype": "int6"},
    torch_dtype=torch.float16,
).cuda()

weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")

# Load unquantized weight and quantize to layer quantization format
layer.load_from_unquantized(weight)
# Transform weight to humming format and prepare default kernels
layer.transform()

# Run quantized GEMM (tuning_config is optional, auto-selected by default)
output = layer(inputs)

print("Quantized GEMM Output:")
print(output)
print("\nReference Output:")
print(inputs.matmul(weight.T))

See docs/config.md for the layer, compute, and tuning configuration reference.

Acknowledgement

This project is highly inspired by

Metadata

Release files for humming-kernels 0.1.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for humming-kernels 0.1.20
File Size Uploaded Provenance Actions
humming_kernels-0.1.20.tar.gz 445.7 kB With provenance Details for humming_kernels-0.1.20.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for humming-kernels 0.1.20
File Actions
humming_kernels-0.1.20-cp311-abi3-win_arm64.whl CPython 3.11 abi3 Windows ARM64 Details for humming_kernels-0.1.20-cp311-abi3-win_arm64.whl
humming_kernels-0.1.20-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for humming_kernels-0.1.20-cp311-abi3-win_amd64.whl
humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl

Total release size: 2.8 MB

Release files / humming_kernels-0.1.20.tar.gz

Download URL humming_kernels-0.1.20.tar.gz
Size 445.7 kB
Tags Source
SHA-256 checksum
How to use checksums
40c1fc66609bef617dc09ff8b9e32bde08db1c58f85a0e97a604f8995b5f838b
BLAKE2b-256 checksum
How to use checksums
62bc79507555b8a55e11b0b56fe83c91d7a2aaa9912bd4949ab45d0731abfdb1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / humming_kernels-0.1.20-cp311-abi3-win_arm64.whl

Download URL humming_kernels-0.1.20-cp311-abi3-win_arm64.whl
Size 558.0 kB
Tags CPython 3.11 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
e65b086862c578cd32b45d75bf1013aafb728be583bf62051cf875c2cb92901a
BLAKE2b-256 checksum
How to use checksums
ee288dec499ca5e17fddb9db7e56e9df2b6b279ad85b566bb6b65574bdc1aafe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / humming_kernels-0.1.20-cp311-abi3-win_amd64.whl

Download URL humming_kernels-0.1.20-cp311-abi3-win_amd64.whl
Size 567.2 kB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
3b75e4279ea5f0d4887fc344c98aca1bb69522d86ec3850a0cb77749524fa2c7
BLAKE2b-256 checksum
How to use checksums
be7a0e9f00cd824863f8f0c9a6680effc9afe7a042f1af93d43be6607e9fb588
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
Size 616.4 kB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
d4747cae223f81bb8d18bee766a5f41e67484a65bd665534fdbc8dc7adabd7c1
BLAKE2b-256 checksum
How to use checksums
53b519eaeed76d26248b8fe9276d2f9a4346e250fd7d2cfb0784ae4929d7efcc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL humming_kernels-0.1.20-cp310-abi3-manylinux_2_28_aarch64.whl
Size 601.9 kB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
1c7ecbeacb8f449136140a7fea68be06f2e5ce083e52de6b2e37e91158c3eee3
BLAKE2b-256 checksum
How to use checksums
07111d3961268098460ea32685ab687bdac1b6bbf681a488b82ef5a5098d872b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for humming-kernels 0.1.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 5 files in this release have been uploaded with provenance metadata.

Select from 5 files

Attestation for humming_kernels-0.1.20.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page