Humming

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference.

Key Features

High Flexibility
- Supports inference for any weight type under 8-bit across FP16 / BF16 / FP8 / FP4 / INT8 / INT4 activations (provided the activation's dynamic range covers the weight type).
- Supports various quantization strategies.
- Supports various scale types (BF16, FP16, E4M3, E5M2, and UE8M0).
- Supports both Dense GEMM and MoE GEMM.
High Compatibility: supports all NVIDIA GPUs from SM75+ (Turing architecture) and beyond.
High Performance
- Delivers State-of-the-Art (SOTA) throughput and efficiency across a wide range of computational scenarios.
Ultra-Lightweight
- Minimal dependencies: Requires only PyTorch and NVCC.
- Compact footprint: The package size is only 100+KB.

Support Matrix

Activation Type	Supported Devices	Supported Weight Types
FP16 (e5m10)	SM75+	• Symmetric INT1-8 • INT1-8 with dynamic zero point • Arbitrary signed FP (kBits ≤ 8, kExp ≤ 5)
BF16 (e8m7)	SM80+	• Symmetric INT1-8 • INT1-8 with dynamic zero point • Arbitrary signed FP (kBits ≤ 8)
FP8 (e4m3)	SM89+	• Symmetric INT1-5 • INT1-4 with dynamic zero point • Arbitrary signed FP (kExp ≤ 4, kMan ≤ 3)
FP8 (e5m2)	SM89+	• Symmetric INT1-4 • INT1-3 with dynamic zero point • Arbitrary signed FP (kExp ≤ 5, kMan ≤ 2)
FP4 (e2m1)	SM120+	• Symmetric INT1-3 • INT1-2 with dynamic zero point • Arbitrary signed FP (kExp ≤ 2, kMan ≤ 1)
INT8	SM75+	• Symmetric INT1-8 • INT1-7 with dynamic zero point
INT4	SM80+	• Symmetric INT1-4 • INT1-3 with dynamic zero point

Getting Started

Installation

pip install git+https://github.com/inclusionAI/humming.git

Usage Example

import torch
from humming.layer import HummingLayer

layer = HummingLayer(
    shape_n=8192,
    shape_k=8192,
    weight_config={"dtype": "int6"},
    torch_dtype=torch.float16,
).cuda()

weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")

# Load unquantized weight and quantize to layer quantization format
layer.load_from_unquantized(weight)
# Transform weight to humming format and prepare default kernels
layer.transform()

# Run quantized GEMM (tuning_config is optional, auto-selected by default)
output = layer(inputs)

print("Quantized GEMM Output:")
print(output)
print("\nReference Output:")
print(inputs.matmul(weight.T))

Acknowledgement

This project is highly inspired by

DeepGEMM
Marlin Kernel and vLLM Marlin Kernel
lmdeploy GEMM kernel
CUTLASS

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

humming_kernels-0.1.11.tar.gz (242.4 kB view details)

Uploaded Jul 15, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

humming_kernels-0.1.11-py3-none-any.whl (205.5 kB view details)

Uploaded Jul 15, 2026 Python 3

File details

Details for the file humming_kernels-0.1.11.tar.gz.

File metadata

Download URL: humming_kernels-0.1.11.tar.gz
Upload date: Jul 15, 2026
Size: 242.4 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for humming_kernels-0.1.11.tar.gz
Algorithm	Hash digest
SHA256	`756ca343c665ba127655c1abd7450bfbc6999b7fe012e10159ebe520e1f2651a`
MD5	`de000803531159b10622fc25fa7b4e5d`
BLAKE2b-256	`897045c7af4d238676abf0e6066a8e39533412e36573b65481a41643cc775fc9`

See more details on using hashes here.

Provenance

The following attestation bundles were made for humming_kernels-0.1.11.tar.gz:

Publisher: publish.yml on inclusionAI/humming

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: humming_kernels-0.1.11.tar.gz
- Subject digest: 756ca343c665ba127655c1abd7450bfbc6999b7fe012e10159ebe520e1f2651a
- Sigstore transparency entry: 2172962884
- Sigstore integration time: Jul 15, 2026
Source repository:
- Permalink: inclusionAI/humming@32ca4bcc70cc3115a0990cb40ea4957f3dc69dc2
- Branch / Tag: refs/tags/v0.1.11
- Owner: https://github.com/inclusionAI
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@32ca4bcc70cc3115a0990cb40ea4957f3dc69dc2
- Trigger Event: release