qnnx-backend

qnnx-backend is an experimental PyTorch torch.compile backend for Qualcomm QNN.

It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.

[!WARNING] qnnx-backend is currently experimental and intended for inference only.

Features

• PyTorch integration through torch.compile(..., backend="qnn")
• Qualcomm NPU/HTP acceleration
• Qualcomm GPU acceleration
• Automatic NPU → GPU → CPU fallback
• Explicit device selection
• Operator compatibility validation
• Static model parameter and buffer freezing
• QNN availability detection

Requirements

• Windows ARM64
• Qualcomm Snapdragon platform with QNN support
• Python 3.13

Tested configuration

• PyTorch 2.13
• ONNX Runtime QNN 2.4
• Snapdragon X Elite

Installation

qnnx-backend currently requires Python 3.13 ARM64 on Windows ARM64.

Important: make sure you are using a native ARM64 Python installation.
A Python x64 installation will not use the required Windows ARM64 wheels.

Python.org provides a dedicated Windows ARM64 installer for Python 3.13.

You can verify your Python architecture with:

python -c "import platform; print(platform.machine())"

The output should be:

ARM64

Then install PyTorch:

python -m pip install torch==2.13.0+cpu --index-url https://download.pytorch.org/whl/cpu

Finally, install qnnx_backend:

python -m pip install qnnx-backend

Usage

Basic usage

qnnx_backend integrates with PyTorch through torch.compile.

import torch
import qnnx_backend

model = MyModel().eval()
x = torch.randn(1, 32)

compiled_model = torch.compile(
    model,
    backend="qnn",
)

with torch.inference_mode():
    output = compiled_model(x)

By default, qnnx_backend automatically selects an available QNN accelerator.

Checking QNN availability

You can check whether at least one QNN accelerator is available:

qnnx_backend.is_available()

You can also check a specific device:

qnnx_backend.is_available("npu")
qnnx_backend.is_available("gpu")

Device selection

The QNN backend supports the following device options:

• "auto" — automatically selects an available accelerator
• "npu" — uses the Qualcomm NPU/HTP backend
• "gpu" — uses the Qualcomm GPU backend
• "cpu" — uses the original PyTorch graph without QNN acceleration

The default is "auto".

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
    },
)

Backend options

Option	Values	Default	Description
device	auto, npu, gpu, cpu	auto	Selects the execution backend.
strict	True, False	False	Controls whether compilation failures can fall back.
fp16	True, False	False	Enables HTP FP16 precision for NPU execution.

Automatic fallback

With the default configuration, qnnx_backend falls back when QNN compilation is not possible.

In "auto" mode, devices are attempted in the following order:

NPU -> GPU -> CPU

For example:

compiled_model = torch.compile(
    model,
    backend="qnn",
)

If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.

Strict mode

Use strict=True when execution on the selected QNN device is required:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "strict": True,
    },
)

In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.

This is useful when testing whether a graph is actually supported by a specific QNN backend.

NPU FP16 execution

For NPU execution, FP16 precision can be enabled with:

compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "fp16": True,
    },
)

This option controls the QNN HTP FP16 precision setting.

Inference

qnnx_backend currently targets inference workloads.

Models should be switched to evaluation mode before compilation:

model.eval()

Inference should normally be performed using:

with torch.inference_mode():
    output = compiled_model(x)

Training and backward execution are not currently supported.

Model parameters and buffers

Model parameters and buffers are captured when the QNN graph is compiled.

For example:

compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

compiled_model(x)

After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.

Recompile the model after changing its parameters:

compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

Static shapes

Version 0.1.0 currently targets static input shapes.

Compile the model using the input shapes that will be used during inference.

Dynamic-shape support is not currently provided.

Operator support

qnnx_backend validates exported ONNX operators against the currently supported QNN backends.

Operator support may differ between NPU/HTP and GPU.

See the operator compatibility list for the current support matrix.

Benchmarks

Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.

The results below show steady-state inference latency after warm-up.

Small models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

Model	Batch	PyTorch CPU	QNN GPU	QNN NPU
Small MLP	16	0.042 ms	0.201 ms	0.203 ms
Small CNN	16	0.299 ms	0.953 ms	0.212 ms
Small Transformer	16	0.750 ms	1.459 ms	0.306 ms

Medium models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

Model	Batch	PyTorch CPU	QNN GPU	QNN NPU
Medium MLP	16	2.523 ms	0.505 ms	0.333 ms
Medium CNN	16	134.117 ms	11.292 ms	2.143 ms
Medium Transformer	16	47.343 ms	9.176 ms	2.036 ms

See detailed benchmarks for batch-size scaling, first-call latency, median, p95, standard deviation, and benchmark methodology.

Author

Developed by Thomas Le Gall — ORCID · GitLab.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Copyright © 2026 Thomas Le Gall.