cuDNN Frontend (FE)

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels — scaled dot-product attention (SDPA / Flash Attention), grouped GEMM fusions for Mixture-of-Experts (MoE) training, fused normalization + activation, and more.

It provides a header-only C++ API and a Python interface (with native PyTorch integration) to the cuDNN Graph API, targeting NVIDIA Hopper (H100/H200) and Blackwell (B200/GB200/GB300) GPUs across FP16, BF16, FP8, and MXFP8 precision.

Links: Documentation · Blog & Deep Dives · PyPI · Release Notes · Samples

🚀 Latest news:

We will begin open-sourcing kernels based on customer needs, with the goal to educate developers and enable them to customize as needed.

We are now shipping OSS kernels, allowing you to inspect, modify, and contribute to the core logic. Check out our latest implementations:

• FROST GEMM engine: JIT-compiled Blackwell GEMM engine reachable through the ordinary cudnn.pygraph API — matmul, grouped (MoE) matmul, block-scaled FP4/FP8, and chained pointwise epilogues are fused into one kernel from the graph you already built. Opt in with CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1; it is then a candidate for every matmul graph it can serve, ranked against the backend's own plans.
• GEMM + Amax: Optimized FP8 matrix multiplication with absolute maximum calculation.
• GEMM + SwiGLU: High-performance implementation of the SwiGLU activation fused with GEMM.
• GEMM + sReLU: High-performance implementation of squared-ReLU fused with GEMM.
• GEMM + dsReLU: High-performance implementation of dsquared-ReLU fused with GEMM.
• Grouped GEMM (BF16): Unfused BF16 grouped GEMM with dense and discrete MoE weight layouts.
• Grouped GEMM + GLU: Unified BF16 and legacy block-scaled grouped GEMM GLU API supporting dense and discrete MoE weight layouts.
• Grouped GEMM + GLU + Hadamard: Dense grouped GEMM GLU forward fusion with a fused Hadamard transform and per-expert AMAX reduction.
• Grouped GEMM + dGLU: Unified BF16 and legacy block-scaled grouped GEMM dGLU backward API supporting dense and discrete MoE weight layouts.
• Grouped GEMM + SwiGLU: SwiGLU activation fused with Grouped GEMM.
• Grouped GEMM + dSwiglu: dSwiglu activation fused with Grouped GEMM.
• Grouped GEMM + sReLU: Contiguous grouped squared-ReLU GEMM for MoE workloads.
• Grouped GEMM + dsReLU: Contiguous grouped dsquared-ReLU GEMM for MoE workloads.
• Discrete Grouped GEMM + SwiGLU: Per-expert-pointer SwiGLU grouped GEMM for MoE workloads without weight packing.
• Discrete Grouped GEMM + dSwiGLU: Per-expert-pointer dSwiGLU backward grouped GEMM for MoE workloads without weight packing.
• Grouped GEMM + Quant: Legacy dense-only grouped GEMM quant API for MoE FC2/dFC1 workloads.
• Grouped GEMM + Quant (Unified): Unified grouped GEMM quant API with per-row gating for MoE FC2/dFC1 workloads.
• Grouped GEMM + Wgrad: Unified BF16 and legacy block-scaled grouped GEMM weight-gradient API supporting dense and discrete output layouts for MoE workloads.
• BSA: Block-sparse attention forward and backward CuTe DSL kernels for block-level routing metadata.
• NSA: Native Sparse attention as described in the Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention.
• SDPA Backward: SM100, D=256: SDPA Backward pass for D=256 on SM100.
• cudnn SDPA Fprop: Open sourcing the Hopper and Blackwell fprop kernels with stats.
• Fused RMSNorm + SiLU: Implementation of a fused kernel of RMS normalization followed by SiLU (Swish) activation.
• SDPA PyTorch Op: PyTorch custom operator for cuDNN-accelerated Scaled Dot-Product Attention with autograd and torch.compile support.
• DSA: DSA/CSA kernels for DSv4 and DSv3.2 for fprop and bprop.

Contributor credits for these OSS CuTe DSL kernels are listed in Acknowledgements.

Tech talks

• See our latest talk on GPU-Mode

  ▶ Watch on YouTube

🔥🔥🔥 SOTA Attention Kernels from cudnn backend

Llama 3.1 style Forward and Bprop with causal masking (GB300)

Deepseek v3 style Forward and Bprop with causal masking (GB300)

Key Features

• Unified Graph API: Create reusable, persistent cudnn_frontend::graph::Graph objects to describe complex subgraphs.
• Ease of Use: Simplified C++ and Python bindings (via pybind11) that abstract away the boilerplate of the backend API.
• Performance: Built-in autotuning and support for the latest NVIDIA GPU architectures.

Installation

🐍 Python

The easiest way to get started is via pip:

pip install nvidia-cudnn-frontend

Requirements:

• Python 3.9+
• NVIDIA driver and CUDA Toolkit
• NVIDIA cuDNN (minimum 8.5.0)

⚙️ C++ (Header Only)

Since the C++ API is header-only, integration is seamless. Simply include the header in your compilation unit:

#include <cudnn_frontend.h>

Ensure your include path points to the include/ directory of this repository.

Building from Source

If you want to build the Python bindings from source or run the C++ samples:

1. Dependencies

• python-dev (e.g., apt-get install python-dev)
• Dependencies listed in requirements.txt (pip install -r requirements.txt)

2. Python Source Build

pip install -v git+https://github.com/NVIDIA/cudnn-frontend.git

Environment variables CUDAToolkit_ROOT and CUDNN_PATH can be used to override default paths.

3. C++ Samples Build

mkdir build && cd build
cmake -DCUDNN_PATH=/path/to/cudnn -DCUDAToolkit_ROOT=/path/to/cuda ../
cmake --build . -j16
./bin/samples

Documentation & Examples

• Developer Guide: Official NVIDIA Documentation (latest)
• Blog & Deep Dives: nvidia.github.io/cudnn-frontend — release notes, installation guides, and technical deep-dives (MXFP8 attention, FP8 scale layouts, etc.)
• C++ Samples: See samples/cpp for end-to-end examples covering convolution, matmul, SDPA / Flash Attention, normalization, and more.
• Python Samples: See samples/python for Jupyter notebooks and PyTorch integration patterns.
• OSS Kernels: See python/cudnn/ for source of SDPA, grouped GEMM + SwiGLU/GLU, RMSNorm + SiLU, Native Sparse Attention, and other open-sourced kernels.
• PyTorch Custom Ops: See python/cudnn/experimental/ops for torch.compile-compatible wrappers around cuDNN kernels.

🤝 Contributing

We strictly welcome contributions! Whether you are fixing a bug, improving documentation, or optimizing one of our new OSS kernels, your help makes cuDNN better for everyone.

1. Check the Contribution Guide for details.
2. Fork the repo and create your branch.
3. Submit a Pull Request.

Debugging

To view the execution flow and debug issues, you can enable logging via environment variables:

# Log to stdout
export CUDNN_FRONTEND_LOG_INFO=1
export CUDNN_FRONTEND_LOG_FILE=stdout

# Log to a file
export CUDNN_FRONTEND_LOG_INFO=1
export CUDNN_FRONTEND_LOG_FILE=execution_log.txt

Logging Levels:

• CUDNN_FRONTEND_LOG_INFO=0: No logging
• CUDNN_FRONTEND_LOG_INFO=1: Full logging with tensor dumps
• CUDNN_FRONTEND_LOG_INFO=10: Basic logging (safe for CUDA graph capture)

Alternatively, you can control logging programmatically via cudnn_frontend::isLoggingEnabled().

OSS engine selection:

The open-source engines are opt-in while they mature: set the flag below and they become candidates for every graph they can serve, ranked against the cuDNN backend's own engines in one list. Engines that are the only implementation of their operation (GDN/KDA) need no flag.

# Offer the maturing open-source engines (FROST GEMM / SDPA) as plan candidates.
export CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1

# Where JIT-compiled kernels are cached (default: $XDG_CACHE_HOME/cudnn_gemm/kernel_cache).
export CUDNN_FRONTEND_GEMM_KERNEL_CACHE=/path/to/cache

graph.plans is the ranked list and graph.get_plan_name_at_index(i) names each entry — an OSS engine reports its engine name, the backend reports the backend plan name. Pin one with graph.select_plan(i) (strict: that plan runs or the build fails) and exclude by name with the classic graph.deselect_engines([...]). graph.selected_engine is the engine that ran, or None when the backend served the graph.

Environment report

When filing a bug, include the output of the environment collector — it reports the frontend/backend versions, GPU/driver properties, and every cuDNN/CUDA library copy on the system (loaded vs on disk):

python -m cudnn.collect_env

If import cudnn itself fails, download collect_env.py and run it standalone with any Python.

Overriding the CUDA runtime library

When the frontend is built with dynamic loading enabled, it locates the CUDA runtime (libcudart.so.*) at runtime by searching for the supported major versions. In some environments (for example, containers such as GKE where the TCPXO NCCL plugin mounts a different libcudart major version from the host) multiple versions of libcudart may be visible on the library search path, and the automatic detection aborts with a Multiple libcudart libraries found error.

To resolve this, set the CUDNN_FRONTEND_CUDART_LIB_NAME environment variable to the library name (or full path) that should be loaded. This bypasses the automatic detection:

export CUDNN_FRONTEND_CUDART_LIB_NAME=libcudart.so.13
# or an absolute path
export CUDNN_FRONTEND_CUDART_LIB_NAME=/usr/local/cuda/lib64/libcudart.so.13

License

This project is distributed primarily under the Apache License 2.0. A subset of files remain under the MIT License; each source file declares its license with an SPDX SPDX-License-Identifier: tag. See LICENSING.md for the full list of MIT-licensed files and the rationale, and THIRD_PARTY_LICENSES.txt for third-party attributions.