SnowLLM

A local LLM inference framework, hand-tuned for the Ryzen AI Max APU.

SnowLLM is fast with:

• Kernels hand-tuned for the Ryzen AI Max series
• Compute precision chosen for what the hardware runs fastest
• Tuned for low-concurrency inference
• Speculative decoding

Demo

https://github.com/user-attachments/assets/fa191709-bca4-4caa-a5ed-9a0ef290a76b

SnowLLM vs. llama.cpp on a single Ryzen AI Max+ 395, both driven through pi.dev on nginx log triage task, running Qwen3.6 35B A3B with a 2.4x speedup!

_{Qwen3.6-35B-A3B UD-Q4_K_XL GGUF for both, on the same hardware, one engine at a time.}

Performance

Measured on one Ryzen AI Max+ 395. Every model, precision, drafter and context length, with the runs behind each number: https://snowllm.dev/#benchmarks.

Accuracy: 91.7% avg@4 on AIME 2026 (±4.2), vs. Qwen's reported 92.7 at avg@8.

Support matrix

Tested only on the Ryzen AI Max+ 395; the rest of the Ryzen AI Max 300 and 400 series (same gfx1151) should work but unverified.

Hugging Face safetensors at FP8, and GGUF K-quants and i-quants. snowllm recipes lists the lot; a recipe that names a snowllm newer than yours says so instead of serving:

qwen3.6-35b-a3b-q4-k-xl is the 35B MoE as a GGUF, 24.2 GiB resident, for a machine that cannot hold the FP8 file. Where a drafter ships beside the weights there is a -dflash or -dflash2 recipe of its own, and qwen3.6-35b-a3b-fp8-mtp is the FP8 weights decoding through the MTP head that checkpoint already carries.

Resident is weights only; the KV pool is what is left under --gpu-memory-utilization. The DeepSeek wants a 128 GiB machine with a large carve-out, and --device-map auto at long context. Flash-Next wants the same machine and no --device-map: another 26.8 GiB of it is a lookup table read off disk per batch, so keep it on the SSD you serve from. Only the Qwens take image input.

Requirements

• Linux x86_64 or Windows 11, AMD gfx1151 (Ryzen AI Max 300 and 400 series — Strix Halo and Gorgon Halo). WSL2 works; ROCm reaches the GPU there through /dev/dxg, so set it up with AMD's WSL guide first
• ROCm 7.x.
• Python 3.10 – 3.14, and ROCm builds of torch and torchvision — see below.

Install

curl -fsSL https://snowllm.dev/install.sh | sh

• Checks the machine before downloading anything: GPU, driver access, Python, disk, network
• Installs into ~/.local/share/snowllm, links snowllm into ~/.local/bin
• Re-run to upgrade; sh -s -- --uninstall to remove

Or do it by hand — torch and torchvision must come from AMD's index:

# https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html
pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
    "torch[device-gfx1151]==2.12.0+rocm7.14.0" \
    "torchvision==0.27.0+rocm7.14.0"
pip install snowllm snowllm-kernels

Integrations

snowllm pi points Pi at a local server by writing every model to ~/.pi/agent/models.json.

snowllm pi --port 8000                        # a server on this machine
snowllm pi --host 192.168.1.10 --port 8000    # one elsewhere on the network

--host is where Pi should reach the server, localhost by default; --port is the same 8000 the server listens on.

Get a model

A recipe is one model at one precision. snowllm pull fetches it:

snowllm recipes                      # what there is
snowllm pull qwen3.6-35b-a3b-fp8     # or just `snowllm pull`, and pick from the list

• Fetches over several connections at once, and resumes where it stopped
• Checks every file against the sha256 the repository publishes
• A recipe that wants a newer snowllm downloads anyway and says which version it wants

SNOWLLM_MODELS is where checkpoints live — ~/models by default. pull writes there, and snowllm <recipe> looks there; --dir overrides it per command.

Or fetch it yourself. A local Hugging Face safetensors directory is all snowllm wants:

hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8

A GGUF works too — snowllm pull qwen3.6-35b-a3b-q4-k-xl-dflash gets the UD-Q4_K_XL quant with its DFlash drafter beside it: a smaller download, and llama.cpp serves the same file.

Run

snowllm qwen3.6-35b-a3b-fp8          # a recipe you pulled, or a path
snowllm ~/models/Qwen3.6-35B-A3B-FP8

Common flags (snowllm --help lists every option):

Roadmap

• More models
• Tensor parallelism at any degree
• NPU/GPU co-working prefill
• Native INT4 quantization

Questions

Ask in Discussions. Hardware reports from the rest of the Ryzen AI Max 300 and 400 series are especially useful — only the 395 has been tested here.

License

• SnowLLM: Apache-2.0
• SnowLLM-Kernels: All rights reserved