Skip to main content

FlashInfer

High-Performance GPU Kernels for Inference

| Documentation | Latest Release | Blog | Slack | Discussion Forum |

Build Status Documentation

FlashInfer is a library and kernel generator for inference that delivers state-of-the-art performance across diverse GPU architectures. It provides unified APIs for attention, GEMM, and MoE operations with multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM.

Why FlashInfer?

  • State-of-the-art Performance: Optimized kernels for prefill, decode, and mixed batching scenarios
  • Multiple Backends: Automatically selects the best backend for your hardware and workload
  • Modern Architecture Support: Support for SM75 (Turing) and later (through Blackwell)
  • Low-Precision Compute: FP8 and FP4 quantization for attention, GEMM, and MoE operations
  • Production-Ready: CUDAGraph and torch.compile compatible for low-latency serving

Core Features

Attention Kernels

  • Paged and Ragged KV-Cache: Efficient memory management for dynamic batch serving
  • Decode, Prefill, and Append: Optimized kernels for all attention phases
  • MLA Attention: Native support for DeepSeek's Multi-Latent Attention
  • Cascade Attention: Memory-efficient hierarchical KV-Cache for shared prefixes
  • Sparse Attention: Block-sparse and variable block-sparse patterns
  • POD-Attention: Fused prefill+decode for mixed batching

GEMM & Linear Operations

  • BF16 GEMM: BF16 matrix multiplication for SM10.0+ GPUs.
  • FP8 GEMM: Per-tensor and groupwise scaling
  • FP4 GEMM: NVFP4 and MXFP4 matrix multiplication for Blackwell GPUs
  • Grouped GEMM: Efficient batched matrix operations for LoRA and multi-expert routing

Mixture of Experts (MoE)

  • Fused MoE Kernels
  • Multiple Routing Methods: DeepSeek-V3, Llama-4, and standard top-k routing
  • Quantized MoE: FP8 and FP4 expert weights with block-wise scaling

Sampling & Decoding

  • Sorting-Free Sampling: Efficient Top-K, Top-P, and Min-P without sorting
  • Speculative Decoding: Chain speculative sampling support

Communication

  • AllReduce: Custom implementations
  • Multi-Node NVLink: MNNVL support for multi-node inference
  • NVSHMEM Integration: For distributed memory operations

Other Operators

  • RoPE: LLaMA-style rotary position embeddings (including LLaMA 3.1)
  • Normalization: RMSNorm, LayerNorm, Gemma-style fused operations
  • Activations: SiLU, GELU with fused gating

GPU Support

Architecture Compute Capability Example GPUs
Turing SM 7.5 T4, RTX 20 series
Ampere SM 8.0, 8.6 A100, A10, RTX 30 series
Ada Lovelace SM 8.9 L4, L40, RTX 40 series
Hopper SM 9.0 H100, H200
Blackwell SM 10.0, 10.3 B200, B300
Blackwell SM 11.0 Jetson Thor
Blackwell SM 12.0, 12.1 RTX 50 series, DGX Spark

Note: Not all features are supported across all compute capabilities.

News

Latest: GitHub Release

Notable updates:

  • [2025-10-08] Blackwell support added in v0.4.0
  • [2025-03-10] Blog Post Sorting-Free GPU Kernels for LLM Sampling, which explains the design of sampling kernels in FlashInfer.

Getting Started

Installation

Quickstart:

pip install flashinfer-python

Package Options:

  • flashinfer-python: Core package that compiles/downloads kernels on first use
  • flashinfer-cubin: Pre-compiled kernel binaries for all supported GPU architectures
  • flashinfer-jit-cache: CUDA-specific shim that installs architecture-specific pre-built kernel providers

For faster initialization and offline usage, install the optional packages to have most kernels pre-compiled:

pip install flashinfer-python
flashinfer download-kernels

flashinfer download-kernels installs the flashinfer-cubin and flashinfer-jit-cache wheels that match the installed flashinfer-python and CUDA versions. By default it installs pre-built kernels for every supported GPU architecture (several GB); pass --mode minimal to install the jit-cache provider only for the visible GPUs, or --sm (e.g. --sm sm90a, repeatable) to choose architectures explicitly.

Upgrading: the kernel wheels must match flashinfer-python exactly and are not upgraded with it. Run flashinfer download-kernels again after every pip install -U flashinfer-python; until then FlashInfer warns at import and ignores the old wheels (falling back to on-demand downloads and JIT compilation). Also note that flashinfer-cubin is no longer published to PyPI (0.6.13 was the last release there): a plain pip install -U flashinfer-cubin installs that stale wheel, so install it through flashinfer download-kernels or from --index-url https://flashinfer.ai/whl. See Upgrading.

For Blackwell (SM100+) CuTe DSL kernels, install with the CUDA 13 extra to enable Blackwell-optimized kernels:

pip install flashinfer-python[cu13]

Verify Installation

flashinfer show-config

Basic Usage

import torch
import flashinfer

# Single decode attention
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)  # [num_qo_heads, head_dim]
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)  # [kv_len, num_kv_heads, head_dim]
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)

output = flashinfer.single_decode_with_kv_cache(q, k, v)

See documentation for comprehensive API reference and tutorials.

Install from Source

git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
cd flashinfer
python -m pip install -v .

For development, install in editable mode:

python -m pip install --no-build-isolation -e . -v

Note: When using --no-build-isolation, pip does not automatically install build dependencies. FlashInfer requires setuptools>=77. If you encounter an error like AttributeError: module 'setuptools.build_meta' has no attribute 'prepare_metadata_for_build_editable', upgrade pip and setuptools first:

python -m pip install --upgrade pip setuptools

Build optional packages:

# flashinfer-cubin
python -m build --no-isolation --wheel flashinfer-cubin
python -m pip install flashinfer-cubin/dist/*.whl

Build one JIT-cache provider for the target GPU, then build a shim that depends on that provider. The example below builds an SM90a provider; both wheels must use the same version settings.

export FLASHINFER_JIT_CACHE_PROVIDER_ARCH=9.0a
python -m build --no-isolation --wheel flashinfer-jit-cache-provider

export FLASHINFER_JIT_CACHE_PROVIDER_ARCHS="9.0a"
python -m build --no-isolation --wheel flashinfer-jit-cache

python -m pip install \
  flashinfer-jit-cache-provider/dist/*.whl \
  flashinfer-jit-cache/dist/*.whl

For more details, see the Install from Source documentation.

Nightly Builds

pip install -U --pre flashinfer-python --index-url https://flashinfer.ai/whl/nightly/ --no-deps
pip install flashinfer-python  # Install dependencies from PyPI
flashinfer download-kernels --nightly

CLI Tools

FlashInfer provides several CLI commands for configuration, module management, and development:

# Verify installation and view configuration
flashinfer show-config

# List and inspect modules
flashinfer list-modules
flashinfer module-status

# Manage artifacts and cache
flashinfer download-kernels      # flashinfer-cubin + flashinfer-jit-cache wheels
flashinfer install-cubin-wheel
flashinfer install-jit-cache-wheel
flashinfer download-cubin
flashinfer clear-cache

# For developers: generate compile_commands.json for IDE integration
flashinfer export-compile-commands [output_path]

For complete documentation, see the CLI reference.

API Logging

FlashInfer provides comprehensive API logging for debugging. Enable it using environment variables:

# Enable logging (levels: 0=off (default), 1=basic, 3=detailed, 5=statistics)
export FLASHINFER_LOGLEVEL=3

# Set log destination (stdout (default), stderr, or file path)
export FLASHINFER_LOGDEST=stdout

For detailed information about logging levels, configuration, and advanced features, see Logging in our documentation.

Custom Attention Variants

Users can customize their own attention variants with additional parameters. For more details, refer to our JIT examples.

CUDA Support

Supported CUDA Versions: 12.9, 13.0, and 13.4 (PyTorch nightly)

Note: FlashInfer strives to follow PyTorch's supported CUDA versions plus the latest CUDA release. CUDA 13.4 wheels are built with the preview toolkit and PyTorch nightly; runtime CI currently covers CUDA 12.9 and 13.0.

Adoption

FlashInfer powers inference in:

Contributing

See CONTRIBUTING.md for the development setup, pull request guidelines (PR template, performance numbers for optimizations), and how CI is triggered. Coding agents pick up the same guidance from AGENTS.md; the review guidance is in docs/code_review_guidance_human.md.

Acknowledgement

FlashInfer is inspired by FlashAttention, vLLM, stream-K, CUTLASS, and AITemplate.

Citation

If you find FlashInfer helpful in your project or research, please consider citing our paper:

@article{ye2025flashinfer,
    title = {FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving},
    author = {
      Ye, Zihao and
      Chen, Lequn and
      Lai, Ruihang and
      Lin, Wuwei and
      Zhang, Yineng and
      Wang, Stephanie and
      Chen, Tianqi and
      Kasikci, Baris and
      Grover, Vinod and
      Krishnamurthy, Arvind and
      Ceze, Luis
    },
    journal = {arXiv preprint arXiv:2501.01005},
    year = {2025},
    url = {https://arxiv.org/abs/2501.01005}
}

Metadata

Release files for flashinfer-python 0.7.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for flashinfer-python 0.7.5
File Size Uploaded Provenance Actions
flashinfer_python-0.7.5.tar.gz 129.8 MB Without provenance Details for flashinfer_python-0.7.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for flashinfer-python 0.7.5
File Interpreter ABI Platform Actions
flashinfer_python-0.7.5-py3-none-any.whl Python 3 none any Details for flashinfer_python-0.7.5-py3-none-any.whl

Total release size: 254.1 MB

Release files / flashinfer_python-0.7.5.tar.gz

Download URL flashinfer_python-0.7.5.tar.gz
Size 129.8 MB
Tags Source
SHA-256 checksum
How to use checksums
822c58b54d3ce10c7a8227c31b52ba519d530acbd777919d74eebefcc904537d
BLAKE2b-256 checksum
How to use checksums
e5f10e397db296102e61aa6b6ddd6f5e33caaf3a2703ccf3481aeb9931208e63
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / flashinfer_python-0.7.5-py3-none-any.whl

Download URL flashinfer_python-0.7.5-py3-none-any.whl
Size 124.3 MB
Tags Python 3
SHA-256 checksum
How to use checksums
f7f3ba5c7e6b437734e9687d59ce6f5258642d5ac17c3ffbda78e290fe650ec3
BLAKE2b-256 checksum
How to use checksums
5a0b3b15144dd4685431a68540cb506d48085da991124c6aa60661187938dda7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for flashinfer-python 0.7.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.5 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page