SnowLLM

A local LLM inference framework, hand-tuned for the Ryzen AI Max APU.

SnowLLM is fast with:

• Kernels hand-tuned for the Ryzen AI Max series
• Compute precision chosen for what the hardware runs fastest
• Tuned for low-concurrency inference
• Speculative decoding

Performance

On one Ryzen AI Max+ 395, Qwen3.6-35B-A3B-FP8, 8K in / 1K out, with speculative decoding (MTP) on:

See BENCHMARK.md for details.

Accuracy: 91.7% avg@4 on AIME 2026 (±4.2), vs. Qwen's reported 92.7 at avg@8.

Support matrix

Tested only on the Ryzen AI Max+ 395; the rest of the Ryzen AI 300 series (same gfx1151) should work but unverified. The only supported model today is Qwen3.6-35B-A3B-FP8.

Requirements

• Linux x86_64, AMD gfx1151 (Ryzen AI Max 300 series — Strix Halo)
• ROCm 7.x.
• Python ≥ 3.10, and a ROCm build of PyTorch — see below.

Install

torch must be installed from AMD's index:

# https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html
pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
    "torch[device-gfx1151]==2.12.0+rocm7.14.0"
pip install snowllm snowllm-kernels

Get a model

snowllm loads a local Hugging Face safetensors checkpoint:

hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8

Run

snowllm ~/models/Qwen3.6-35B-A3B-FP8

Common flags (snowllm --help lists every option):

Roadmap

• More models
• Tensor parallelism at any degree
• Adaptive KV cache pool
• NPU/GPU co-working prefill
• Native INT4 quantization

License

• SnowLLM: Apache-2.0
• SnowLLM-Kernels: All rights reserved