SnowLLM

A local LLM inference framework, hand-tuned for the Ryzen AI Max APU.

SnowLLM is fast with:

• Kernels hand-tuned for the Ryzen AI Max series
• Compute precision chosen for what the hardware runs fastest
• Tuned for low-concurrency inference
• Speculative decoding

News

• 2026-10-12 — v0.3.9 adds a persistent daemon and local Web UI for model management and chat, shared JSON configuration, template-based thinking controls, and request/GPU metrics.
• 2026-10-12 — v0.3.8 fixes Windows weight-loading races, GPU memory budgeting and downloads, with quantized weight shuffle and file-read recovery fixes in SnowLLM-Kernels 0.3.4.
• 2026-08-04 — SnowLLM is now open source and available on PyPI! A local LLM inference framework hand-tuned for AMD Ryzen AI Max APUs.
• 2026-09-09 — v0.3.0 adds GGUF support, DeepSeek-V4-Flash and Qwen3.8 models, DFlash speculative decoding, and recipe-based model downloads.
• 2026-10-03 — New demo: 2.4× faster than llama.cpp on an nginx log triage task through Pi, using the same Qwen3.6-35B-A3B GGUF on a single Ryzen AI Max+ 395.

Demo

https://github.com/user-attachments/assets/fa191709-bca4-4caa-a5ed-9a0ef290a76b

SnowLLM vs. llama.cpp on a single Ryzen AI Max+ 395, both driven through pi.dev on nginx log triage task, running Qwen3.6 35B A3B with a 2.4x speedup!

_{Qwen3.6-35B-A3B UD-Q4_K_XL GGUF for both, on the same hardware, one engine at a time.}

Performance

Measured on one Ryzen AI Max+ 395. Every model, precision, drafter and context length, with the runs behind each number: https://snowllm.dev/#benchmarks.

Accuracy: 91.7% avg@4 on AIME 2026 (±4.2), vs. Qwen's reported 92.7 at avg@8.

Support matrix

Tested only on the Ryzen AI Max+ 395; the rest of the Ryzen AI Max 300 and 400 series (same gfx1151) should work but unverified.

Hugging Face safetensors at FP8, and GGUF K-quants and i-quants. snowllm recipes lists the lot; a recipe that names a snowllm newer than yours says so instead of serving:

qwen3.6-35b-a3b-q4-k-xl is the 35B MoE as a GGUF, 24.2 GiB resident, for a machine that cannot hold the FP8 file. Where a drafter ships beside the weights there is a -dflash or -dflash2 recipe of its own, and qwen3.6-35b-a3b-fp8-mtp is the FP8 weights decoding through the MTP head that checkpoint already carries.

Resident is weights only; the KV pool is what is left under --gpu-memory-utilization. The DeepSeek wants a 128 GiB machine with a large carve-out, and --device-map auto at long context. Flash-Next wants the same machine and no --device-map: another 26.8 GiB of it is a lookup table read off disk per batch, so keep it on the SSD you serve from. Only the Qwens take image input.

Requirements

• Linux x86_64 or Windows 11, AMD gfx1151 (Ryzen AI Max 300 and 400 series — Strix Halo and Gorgon Halo). WSL2 works; ROCm reaches the GPU there through /dev/dxg, so set it up with AMD's WSL guide first
• ROCm 7.x.
• Python 3.10 – 3.14, and ROCm builds of torch and torchvision — see below.

Install

curl -fsSL https://snowllm.dev/install.sh | sh

• Checks the machine before downloading anything: GPU, driver access, Python, disk, network
• Installs into ~/.local/share/snowllm, links snowllm into ~/.local/bin
• Re-run to upgrade; sh -s -- --uninstall to remove

Or do it by hand — torch and torchvision must come from AMD's index:

# https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html
pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
    "torch[device-gfx1151]==2.12.0+rocm7.14.0" \
    "torchvision==0.27.0+rocm7.14.0"
pip install snowllm snowllm-kernels

Integrations

snowllm pi points Pi at a local server by writing every model to ~/.pi/agent/models.json.

snowllm pi --port 8000                        # a server on this machine
snowllm pi --host 192.168.1.10 --port 8000    # one elsewhere on the network

--host is where Pi should reach the server, localhost by default; --port is the same 8000 the server listens on.

OpenAI Chat Completions/Responses and Anthropic Messages APIs support streaming, tool calls, and images on supported models; use http://127.0.0.1:8000/v1 for OpenAI or http://127.0.0.1:8000 for Anthropic.

Get a model

A recipe is one model at one precision. snowllm pull fetches it:

snowllm recipes                      # what there is
snowllm pull qwen3.6-35b-a3b-fp8     # or just `snowllm pull`, and pick from the list

• Fetches over several connections at once, and resumes where it stopped
• Checks every file against the sha256 the repository publishes
• A recipe that wants a newer snowllm downloads anyway and says which version it wants

Checkpoints live in ~/models by default. Set models_dir in the JSON configuration or override it with SNOWLLM_MODELS. snowllm <recipe> looks there; snowllm pull --dir PATH selects a different download directory.

Or fetch it yourself. A local Hugging Face safetensors directory is all snowllm wants:

hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8

A GGUF works too — snowllm pull qwen3.6-35b-a3b-q4-k-xl-dflash gets the UD-Q4_K_XL quant with its DFlash drafter beside it: a smaller download, and llama.cpp serves the same file.

Run

snowllm qwen3.6-35b-a3b-fp8          # a recipe you pulled, or a path
snowllm ~/models/Qwen3.6-35B-A3B-FP8

Common flags (snowllm --help lists every option):

Daemon and Web UI

Start the daemon to download models, load or switch checkpoints, chat, and monitor inference through a local Web UI:

snowllm daemon                 # opens the Web UI at http://127.0.0.1:8001
snowllm daemon --no-browser    # terminal / headless use

The inference API stays at http://127.0.0.1:8000 across model switches. You can also manage models from another terminal:

snowllm load qwen3.6-35b-a3b-fp8 --max-model-len 32k
snowllm status
snowllm unload

Configure model storage and the recipe catalogue in the Web UI's Settings. Settings are saved to ~/.config/snowllm/config.json on Linux (or $XDG_CONFIG_HOME/snowllm/config.json) and %APPDATA%\SnowLLM\config.json on Windows. The same configuration applies to snowllm pull and direct serving.

See the daemon guide for configuration overrides, CLI commands, monitoring, and Web UI details.

Roadmap

• More models
• Tensor parallelism at any degree
• NPU/GPU co-working prefill
• Native INT4 quantization

Questions

Ask in Discussions. Hardware reports from the rest of the Ryzen AI Max 300 and 400 series are especially useful — only the 395 has been tested here.

License

• SnowLLM: Apache-2.0
• SnowLLM-Kernels: All rights reserved