qnnx-backend

qnnx-backend is an experimental PyTorch torch.compile backend for Qualcomm QNN.

It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.

Features

• PyTorch integration through torch.compile(..., backend="qnn")
• Qualcomm NPU/HTP acceleration
• Qualcomm GPU acceleration
• Persistent QNN context caching across Python process restarts
• Automatic NPU → GPU → CPU fallback
• Explicit device selection
• Operator compatibility validation
• Static model parameter and buffer freezing
• QNN availability detection

Requirements

• Windows ARM64
• Qualcomm Snapdragon platform with QNN support
• Python 3.13

Note: Python 3.14 is not currently supported because compatible stable PyTorch wheels for Windows ARM64 are not yet available. Support can be added once PyTorch provides the required ARM64 wheels.

Tested configuration

The following versions are covered by the compatibility test matrix:

• Python 3.13
• PyTorch 2.13 and 2.14
• ONNX Runtime QNN 2.4, 2.5 and 2.6
• Snapdragon X Elite

Installation

qnnx-backend currently supports Python 3.13 ARM64 on Windows ARM64.

Important: make sure you are using a native ARM64 Python installation.
A Python x64 installation will not use the required Windows ARM64 wheels.

Python.org provides dedicated Windows ARM64 installers for supported Python versions.

You can verify your Python architecture with:

python -c "import platform; print(platform.machine())"

The output should be:

ARM64

Then install PyTorch:

python -m pip install torch==2.14.0+cpu --index-url https://download.pytorch.org/whl/cpu

Finally, install qnnx-backend:

python -m pip install qnnx-backend

Usage

Basic usage

qnnx_backend integrates with PyTorch through torch.compile.

import torch
import qnnx_backend

model = MyModel().eval()
x = torch.randn(1, 32)

compiled_model = torch.compile(
    model,
    backend="qnn",
)

with torch.inference_mode():
    output = compiled_model(x)

By default, qnnx_backend automatically selects an available QNN accelerator.

Checking QNN availability

You can check whether at least one QNN accelerator is available:

qnnx_backend.is_available()

You can also check a specific device:

qnnx_backend.is_available("npu")
qnnx_backend.is_available("gpu")

Device selection

The QNN backend supports the following device options:

• "auto" — automatically selects an available accelerator
• "npu" — uses the Qualcomm NPU/HTP backend
• "gpu" — uses the Qualcomm GPU backend
• "cpu" — uses the original PyTorch graph without QNN acceleration

The default is "auto".

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
    },
)

Backend options

Automatic fallback

With the default configuration, qnnx_backend falls back when QNN compilation is not possible.

In "auto" mode, devices are attempted in the following order:

NPU -> GPU -> CPU

For example:

compiled_model = torch.compile(
    model,
    backend="qnn",
)

If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.

Strict mode

Use strict=True when execution on the selected QNN device is required:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "strict": True,
    },
)

In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.

This is useful when testing whether a graph is actually supported by a specific QNN backend.

NPU FP16 execution

For NPU execution, FP16 precision can be enabled with:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "fp16": True,
    },
)

This option controls the QNN HTP FP16 precision setting.

Persistent context cache

QNN graph compilation can introduce a significant startup cost.

Persistent context caching allows qnnx_backend to reuse a previously compiled QNN context across Python process restarts:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "cache": True,
    },
)

On the first execution, the QNN context is compiled and stored on disk. Subsequent executions of the same model and configuration can reuse the cached context instead of compiling it again.

Persistent context caching is supported for both QNN NPU and GPU execution.

A custom cache directory can be specified with:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "cache": True,
        "cache_dir": "./qnn-cache",
    },
)

The cache key includes the exported model and relevant QNN compilation configuration, so incompatible contexts are stored separately.

Inference

qnnx_backend currently targets inference workloads.

Models should be switched to evaluation mode before compilation:

model.eval()

Inference should normally be performed using:

with torch.inference_mode():
    output = compiled_model(x)

Training and backward execution are not currently supported.

Model parameters and buffers

Model parameters and buffers are captured when the QNN graph is compiled.

For example:

compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

compiled_model(x)

After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.

Recompile the model after changing its parameters:

compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

Static shapes

qnnx_backend currently targets static input shapes.

Compile the model using the input shapes that will be used during inference.

Dynamic-shape support is not currently provided.

Operator support

qnnx_backend validates exported ONNX operators against the currently supported QNN backends.

Operator support may differ between NPU/HTP and GPU.

See the operator compatibility list for the current support matrix.

Benchmarks

Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.

The results below show steady-state inference latency after warm-up.

Small models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

Medium models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

Persistent context cache

Persistent context caching also reduces time to first inference across Python process restarts:

Each warm measurement was performed in a fresh Python process using a context generated during the corresponding cold run.

The benchmark documentation provides detailed results for the synthetic architectures above and for three Hugging Face transformer models, together with the methodology, coverage and limitations.

Author

Developed by Thomas Le Gall — ORCID · GitLab.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Copyright © 2026 Thomas Le Gall.