SnowLLM

A local LLM inference framework, hand-tuned for the Ryzen AI Max APU.

For more information, visit snowllm.dev (中文).

SnowLLM is fast with:

• Kernels hand-tuned for the Ryzen AI Max series
• Compute precision chosen for what the hardware runs fastest
• Tuned for low-concurrency inference
• Speculative decoding

News

• 2026-10-19 — GLM-5.3-Flash support with MTP: 26.8 tok/s at 220K context, 4.3× faster than llama.cpp, on a single Ryzen AI Max+ 395 using IQ2_XXS weights. Benchmarks.
• 2026-10-12 — v0.3.9 adds a persistent daemon and local Web UI for model management and chat, shared JSON configuration, template-based thinking controls, and request/GPU metrics.
• 2026-10-12 — v0.3.8 fixes Windows weight-loading races, GPU memory budgeting and downloads, with quantized weight shuffle and file-read recovery fixes in SnowLLM-Kernels 0.3.4.
• 2026-10-03 — New demo: 2.4× faster than llama.cpp on an nginx log triage task through Pi, using the same Qwen3.6-35B-A3B GGUF on a single Ryzen AI Max+ 395.
• 2026-09-09 — v0.3.0 adds GGUF support, DeepSeek-V4-Flash and Qwen3.8 models, DFlash speculative decoding, and recipe-based model downloads.
• 2026-08-04 — SnowLLM is now open source and available on PyPI! A local LLM inference framework hand-tuned for AMD Ryzen AI Max APUs.

Demo

https://github.com/user-attachments/assets/fa191709-bca4-4caa-a5ed-9a0ef290a76b

SnowLLM vs. llama.cpp on a single Ryzen AI Max+ 395, both driven through pi.dev on nginx log triage task, running Qwen3.6 35B A3B with a 2.4x speedup!

_{Qwen3.6-35B-A3B UD-Q4_K_XL GGUF for both, on the same hardware, one engine at a time.}

Performance

Measured on one Ryzen AI Max+ 395. Every model, precision, drafter and context length, with the runs behind each number: https://snowllm.dev/#benchmarks.

Accuracy: 91.7% avg@4 on AIME 2026 (±4.2), vs. Qwen's reported 92.7 at avg@8.

Support matrix

Tested only on the Ryzen AI Max+ 395; validation of other APUs is in progress. The APU target list is gfx1103, gfx1150, gfx1151, gfx1152 and gfx1153. Use the homepage target selector to generate installation commands, or see the APU target table.

Hugging Face safetensors at FP8, and GGUF K-quants and i-quants. snowllm recipes lists the lot; a recipe that names a snowllm newer than yours says so instead of serving:

qwen3.6-35b-a3b-q4-k-xl is the 35B MoE as a GGUF, 24.2 GiB resident, for a machine that cannot hold the FP8 file. Where a drafter ships beside the weights there is a -dflash or -dflash2 recipe of its own, and qwen3.6-35b-a3b-fp8-mtp is the FP8 weights decoding through the MTP head that checkpoint already carries.

Resident is weights only; KV capacity follows the configured context and concurrency. DeepSeek wants a 128 GiB machine with a large carve-out, and an explicit host/disk offload budget at long context. Flash-Next wants the same machine and no --device-map: another 26.8 GiB of it is a lookup table read off disk per batch, so keep it on the SSD you serve from. Only the Qwens take image input.

Requirements

• Linux x86_64 or Windows 11, an AMD Ryzen APU with one of the targets listed above. WSL2 works; ROCm reaches the GPU there through /dev/dxg, so set it up with AMD's WSL guide first
• ROCm 7.x.
• Python 3.10 – 3.14, and ROCm builds of torch and torchvision — see below.

Install

curl -fsSL https://snowllm.dev/install.sh | sh

• Checks the machine before downloading anything: GPU, driver access, Python, disk, network
• Installs into ~/.local/share/snowllm, links snowllm into ~/.local/bin
• Re-run to upgrade; sh -s -- --uninstall to remove

Or do it by hand — torch and torchvision must come from AMD's index:

# https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html
pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
    "torch[device-gfx1151]==2.12.0+rocm7.14.0" \
    "torchvision==0.27.0+rocm7.14.0"
pip install snowllm snowllm-kernels

Integrations

snowllm pi points Pi at a local server by writing every model to ~/.pi/agent/models.json.

snowllm pi --port 8000                        # a server on this machine
snowllm pi --host 192.168.1.10 --port 8000    # one elsewhere on the network

--host is where Pi should reach the server, localhost by default; --port is the same 8000 the server listens on.

OpenAI Chat Completions/Responses and Anthropic Messages APIs support streaming, tool calls, and images on supported models; use http://127.0.0.1:8000/v1 for OpenAI or http://127.0.0.1:8000 for Anthropic.

Web UI

Start the Web UI to download models, load or switch checkpoints, chat, and monitor inference through a local Web UI:

snowllm webui                 # opens the Web UI at http://127.0.0.1:8001
snowllm webui --no-browser    # terminal / headless use

The inference API stays at http://127.0.0.1:8000 across model switches. You can also manage models from another terminal:

snowllm load qwen3.6-35b-a3b-fp8 --max-model-len 32k
snowllm status
snowllm unload

Configure model storage and the recipe catalogue in the Web UI's Settings. Settings are saved to ~/.config/snowllm/config.json on Linux (or $XDG_CONFIG_HOME/snowllm/config.json) and %APPDATA%\SnowLLM\config.json on Windows. The same configuration applies to snowllm pull and direct serving.

See the Web UI guide for configuration overrides, CLI commands, monitoring, and Web UI details.

Get a model

A recipe is one model at one precision. snowllm pull fetches it:

snowllm recipes                      # what there is
snowllm pull qwen3.6-35b-a3b-fp8     # or just `snowllm pull`, and pick from the list

• Fetches over several connections at once, and resumes where it stopped
• Checks every file against the sha256 the repository publishes
• A recipe that wants a newer snowllm downloads anyway and says which version it wants

Checkpoints live in ~/models by default. Set models_dir in the JSON configuration or override it with SNOWLLM_MODELS. snowllm <recipe> looks there; snowllm pull --dir PATH selects a different download directory.

Or fetch it yourself. A local Hugging Face safetensors directory is all snowllm wants:

hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8

A GGUF works too — snowllm pull qwen3.6-35b-a3b-q4-k-xl-dflash gets the UD-Q4_K_XL quant with its DFlash drafter beside it: a smaller download, and llama.cpp serves the same file.

Run

snowllm qwen3.6-35b-a3b-fp8          # a recipe you pulled, or a path
snowllm ~/models/Qwen3.6-35B-A3B-FP8

Common flags (snowllm --help lists every option):

Roadmap

• More models
• Tensor parallelism at any degree
• NPU/GPU co-working prefill
• Native INT4 quantization

Questions

Ask in Discussions. Hardware reports from the rest of the Ryzen AI Max 300 and 400 series are especially useful — only the 395 has been tested here.

License

• SnowLLM: Apache-2.0
• SnowLLM-Kernels: All rights reserved

Changelog / 更新日志

0.3.18 — 2026-10-28

• Fix Flash-Next DFlash2 startup failures. 修复 Flash-Next DFlash2 启动失败的问题。
• Show a one-line summary in update notifications. 更新提示改为显示一行摘要。

0.3.17 — 2026-10-27

• Fix V4.1 DSpark tap buffers for the full decoder replay window and reuse activation workspace after target sampling. 修复 V4.1 DSpark 回放窗口的 tap 缓冲规划，并在主模型采样后复用激活工作区。
• Set both V4.1 recipes to 8192-token prefill chunks and require 0.3.17 for the DSpark recipe. V4.1 普通版和 DSpark 版均默认使用 8192-token 预填充分块，DSpark 方案要求 0.3.17。
• Tighten the V4.1 host memory ceiling to 75% so automatic placement can use smaller weight groups instead of excessive whole-layer offload. V4.1 主机内存上限收紧到 75%，让自动放置选择更小的权重组，避免整层卸载占用过多主机内存。

0.3.16 — 2026-10-27

• Restore a shared KV pool across concurrent requests instead of multiplying context capacity by the request count. 恢复多个并发请求共享 KV 池的容量计算，不再按并发数成倍预留上下文容量。
• Restore the 8192-token default prefill chunk and set it explicitly in Flash-Next recipes, preserving the existing GPU memory utilization limits. 恢复默认 8192-token 预填充分块，并在 Flash-Next 方案中显式指定，保持原有 GPU 显存使用比例。

0.3.15 — 2026-10-27

• Restore the original recipe catalogue for SnowLLM 0.3.13 and use a separate schema-2 catalogue by default in 0.3.15. 恢复 SnowLLM 0.3.13 的原目录格式，0.3.15 默认使用独立的 schema 2 目录。
• Read legacy recipe memory settings, migrate saved default catalogue URLs, and report version requirements before validating future recipe options. 兼容旧 recipe 显存参数，自动迁移已保存的默认目录地址，并在校验未来版本参数前提示版本要求。
• Keep compatibility for 0.3.13 temporarily; a future release will remove compatibility with 0.3.13 and earlier. 暂时保留 0.3.13 兼容支持；后续版本将移除对 0.3.13 及更早版本的兼容支持。

0.3.14 — 2026-10-26

• Add DeepSeek V4.1 Flash text inference, including optional embedded DSpark decoding. 新增 DeepSeek V4.1 Flash 文本推理，支持可选的内置 DSpark 解码。
• Improve prefill and decode efficiency, and fix output agreement between ordinary and speculative decoding at longer contexts. 改进 Prefill 与 Decode 效率，修复长上下文下普通解码与推测解码的输出一致性。
• Add explicit GPU, host and disk memory limits, automatic weight placement and expert disk caching. 新增 GPU、主机和磁盘用量上限、自动权重放置及专家磁盘缓存。
• Fix multipart GGUF reads, loading-time memory accounting, cache lifetimes and Windows memory diagnostics. 修复分片 GGUF 读取、加载阶段内存计量、缓存生命周期和 Windows 内存诊断。
• Show actionable CLI capacity errors and prominent bilingual Web UI failure messages with settings and log shortcuts. CLI 容量报错提供可操作的参数建议，Web UI 新增醒目的双语错误提示及设置、日志入口。
• Show release notes in update notifications and require SnowLLM-Kernels 0.3.9. 更新通知支持显示发布说明，依赖 SnowLLM-Kernels 0.3.9。