Piper Kernels

Quantized attention, linear, and convolution operators for PyTorch inference. Piper Kernels combines compact weight formats and GPU kernels with optional compiler fusions that avoid intermediate tensors and repeated preparation. Portable PyTorch references define the corresponding numerical operations.

Use the operators directly or install quantized weights in PyTorch modules. Applications own model loading, checkpoint naming, and device placement; Piper Offload uses these weight formats for its model runtime.

Installation

Requires Python 3.13 or newer and PyTorch 2.14 or newer.

pip install piper-kernels

The base package requires only PyTorch. Optional extras provide weight-format and accelerator dependencies:

For example, install "piper-kernels[convrot,triton]" for ConvRot INT8 GPU execution. On Linux, install the CUDA or ROCm PyTorch build and its matching Triton first; the extra does not install Linux Triton.

On 64-bit Windows, the triton extra selects Triton 3.8 through triton-windows. Optimized execution needs Windows 10 or 11, a supported GPU with a current driver, and the Visual C++ Redistributable for Visual Studio 2015–2022. For NVIDIA, triton-windows bundles the CUDA toolchain and TinyCC; no separate CUDA toolkit or Visual Studio installation is needed. For AMD, use TheRock's ROCm PyTorch build and its matching GPU device packages; see AMD validation coverage.

Quick start

This runs dense Piper Attention on CUDA when available and on CPU otherwise. Inputs use [batch, heads, sequence, head_dim] layout:

import torch
from piper_kernels import piper_attention

device = "cuda" if torch.cuda.is_available() else "cpu"
query, key, value = (
    torch.randn(1, 2, 128, 64, device=device, dtype=torch.bfloat16) for _ in range(3)
)

with torch.inference_mode():
    output = piper_attention(query, key, value, is_causal=False)

print(output.shape)  # torch.Size([1, 2, 128, 64])

Supported GPUs with the required backend installed use an optimized path; other configurations use the slower reference for the same quantized algorithm. Both approximate floating-point attention, and compiler fusions can change rounding. Measure numerical quality and full-operator latency on your workload; see attention for input and backend support.

Guides

These docs describe the checked-out revision. Use the matching Git tag for an installed release. AGENTS.md gives repository instructions for coding agents; VERSIONING.md covers compatibility and releases, and CHANGELOG.md records changes between releases.

Licensed under the Apache License, Version 2.0.