Humming

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference.

Key Features

  • High Flexibility
    • Supports inference for any weight type under 8-bit across FP16 / BF16 / FP8 / FP4 / INT8 / INT4 activations (provided the activation's dynamic range covers the weight type).
    • Supports various quantization strategies.
    • Supports various scale types (BF16, FP16, E4M3, E5M2, and UE8M0).
    • Supports both Dense GEMM and MoE GEMM.
  • High Compatibility: supports all NVIDIA GPUs from SM75+ (Turing architecture) and beyond.
  • High Performance
    • Delivers State-of-the-Art (SOTA) throughput and efficiency across a wide range of computational scenarios.
  • Ultra-Lightweight
    • Minimal dependencies: Requires only PyTorch and NVCC.
    • Compact footprint: The package size is only 100+KB.

Support Matrix

Activation Type Supported Devices Supported Weight Types
FP16 (e5m10) SM75+ • Symmetric INT1-8
• INT1-8 with dynamic zero point
• Arbitrary signed FP (kBits ≤ 8, kExp ≤ 5)
BF16 (e8m7) SM80+ • Symmetric INT1-8
• INT1-8 with dynamic zero point
• Arbitrary signed FP (kBits ≤ 8)
FP8 (e4m3) SM89+ • Symmetric INT1-5
• INT1-4 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 4, kMan ≤ 3)
FP8 (e5m2) SM89+ • Symmetric INT1-4
• INT1-3 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 5, kMan ≤ 2)
FP4 (e2m1) SM120+ • Symmetric INT1-3
• INT1-2 with dynamic zero point
• Arbitrary signed FP (kExp ≤ 2, kMan ≤ 1)
INT8 SM75+ • Symmetric INT1-8
• INT1-7 with dynamic zero point
INT4 SM80+ • Symmetric INT1-4
• INT1-3 with dynamic zero point

Getting Started

Installation

pip install git+https://github.com/inclusionAI/humming.git

Usage Example

import torch
from humming.layer import HummingLayer

layer = HummingLayer(
    shape_n=8192,
    shape_k=8192,
    weight_config={"dtype": "int6"},
    torch_dtype=torch.float16,
).cuda()

weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")

# Load unquantized weight and quantize to layer quantization format
layer.load_from_unquantized(weight)
# Transform weight to humming format and prepare default kernels
layer.transform()

# Run quantized GEMM (tuning_config is optional, auto-selected by default)
output = layer(inputs)

print("Quantized GEMM Output:")
print(output)
print("\nReference Output:")
print(inputs.matmul(weight.T))

Acknowledgement

This project is highly inspired by

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.13)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for linjinzhen from gravatar.com linjinzhen

Requires

Python >=3.10

Provides Extra

cu12 cu13

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

humming_kernels-0.1.12.tar.gz (259.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl (322.7 kB view details)

Uploaded Python 3manylinux: glibc 2.28+ x86-64

humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl (318.0 kB view details)

Uploaded Python 3manylinux: glibc 2.28+ ARM64

File details

Details for the file humming_kernels-0.1.12.tar.gz.

File metadata

  • Download URL: humming_kernels-0.1.12.tar.gz
  • Upload date:
  • Size: 259.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for humming_kernels-0.1.12.tar.gz
Algorithm Hash digest
SHA256 c96c37c95f74379067d2d2d0c5154d12ee9a8cfa29a7c3370a7f3e34f207ca72
MD5 612979f8678f96dc10fa1f191b34dff6
BLAKE2b-256 93b953797fff4b1b31b4567453a365646d724469b1bb0d76c07f8746617d0cb6

See more details on using hashes here.

Provenance

The following attestation bundles were made for humming_kernels-0.1.12.tar.gz:

Publisher: publish.yml on inclusionAI/humming

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 cd3ef712a93f3a9075ea99de2c72bcd3ec89dab3759b3a248d869f5507b60331
MD5 7566d9eb65aaf72ed0d5423f998e2407
BLAKE2b-256 eca2f96912ee7d68f56e61c15555657ffa057b8afbaa21241c63342b5969bb74

See more details on using hashes here.

Provenance

The following attestation bundles were made for humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl:

Publisher: publish.yml on inclusionAI/humming

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 988e8b9da41e3679ae2816bdfb51acab53c05cc56254b5ca98e847f4efcf24fd
MD5 e8db20f0d7d521f46bd70d3b148f4032
BLAKE2b-256 580f01f871dc26f8d7e1683df9464890d48cfd5c4e46c4a264d7ee0868749197

See more details on using hashes here.

Provenance

The following attestation bundles were made for humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl:

Publisher: publish.yml on inclusionAI/humming

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.1.13

3 files

This release

0.1.12 This release

3 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page