Humming

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference.

Key Features

• High Flexibility
  • Supports inference for any weight type under 8-bit across FP16 / BF16 / FP8 / FP4 / INT8 / INT4 activations (provided the activation's dynamic range covers the weight type).
  • Supports various quantization strategies.
  • Supports various scale types (BF16, FP16, E4M3, E5M2, and UE8M0).
  • Supports both Dense GEMM and MoE GEMM.
• High Compatibility: supports all NVIDIA GPUs from SM75+ (Turing architecture) and beyond.
• High Performance
  • Delivers State-of-the-Art (SOTA) throughput and efficiency across a wide range of computational scenarios.
• Ultra-Lightweight
  • Minimal dependencies: Requires only PyTorch and NVCC.
  • Compact footprint: The package size is only 100+KB.

Support Matrix

Getting Started

Installation

pip install tokenspeed-humming

Usage Example

import torch
from humming.layer import HummingLayer

layer = HummingLayer(
    shape_n=8192,
    shape_k=8192,
    weight_config={"dtype": "int6"},
    torch_dtype=torch.float16,
).cuda()

weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")

# Load unquantized weight and quantize to layer quantization format
layer.load_from_unquantized(weight)
# Transform weight to humming format and prepare default kernels
layer.transform()

# Run quantized GEMM (tuning_config is optional, auto-selected by default)
output = layer(inputs)

print("Quantized GEMM Output:")
print(output)
print("\nReference Output:")
print(inputs.matmul(weight.T))

Acknowledgement

This project is highly inspired by

• DeepGEMM
• Marlin Kernel and vLLM Marlin Kernel
• lmdeploy GEMM kernel
• CUTLASS