SnowLLM
A local LLM inference framework, hand-tuned for the Ryzen AI Max APU.
SnowLLM is fast with:
- Kernels hand-tuned for the Ryzen AI Max series
- Compute precision chosen for what the hardware runs fastest
- Tuned for low-concurrency inference
- Speculative decoding
Performance
On one Ryzen AI Max+ 395,
Qwen3.6-35B-A3B-FP8, 8K in / 1K out, with
speculative decoding (MTP) on:
| Users | Context | Output tok/s | Prefill tok/s | TTFT |
|---|---|---|---|---|
| 1 | 8K | 76.7 | 3467 | 2.1 s |
| 4 | 8K | 176.9 | 3430 | 5.3 s |
| 1 | 200K | 49.3 | 1227 | 168.4 s |
See BENCHMARK.md for details.
Accuracy: 91.7% avg@4 on AIME 2026 (±4.2), vs. Qwen's reported 92.7 at avg@8.
Support matrix
Tested only on the Ryzen AI Max+ 395;
the rest of the Ryzen AI 300 series (same gfx1151) should work but unverified. The
only supported model today is Qwen3.6-35B-A3B-FP8.
Requirements
- Linux x86_64, AMD gfx1151 (Ryzen AI Max 300 series — Strix Halo)
- ROCm 7.x.
- Python 3.10 – 3.14, and ROCm builds of torch and torchvision — see below.
Install
curl -fsSL https://raw.githubusercontent.com/SnowLLM/SnowLLM/main/install.sh | sh
- Checks the machine before downloading anything: GPU, driver access, Python, disk, network
- Installs into
~/.local/share/snowllm, linkssnowllminto~/.local/bin - Re-run to upgrade;
sh -s -- --uninstallto remove
Or do it by hand — torch and torchvision must come from AMD's index:
# https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html
pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
"torch[device-gfx1151]==2.12.0+rocm7.14.0" \
"torchvision==0.27.0+rocm7.14.0"
pip install snowllm snowllm-kernels
torchvision is not optional: the supported model takes image input, and its processor fails to load without it.
Get a model
snowllm loads a local Hugging Face safetensors checkpoint:
hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8
Run
snowllm ~/models/Qwen3.6-35B-A3B-FP8
Common flags (snowllm --help lists every option):
| Flag | What it does |
|---|---|
--max-num-seqs |
Concurrency ceiling, up to 256. A memory choice, not a kernel limit: each request pins linear-attention state for its whole life. |
--max-model-len |
Context length, e.g. 32k. A per-request ceiling, not a reservation: the KV pool is shared, and a request that outgrows it is preempted and re-prefilled. |
--gpu-memory-utilization |
Fraction of the GPU the server may occupy, 0.9 by default. The KV pool is whatever is left under it once everything else is down. |
--prefix-cache-gib |
Host RAM for remembering shared prompt prefixes, 4 by default, 0 to disable. A repeat behind a 16K prefix costs 0.36 s instead of 5.1 s; the first one pays ~1%. |
--max-num-batched-tokens |
Tokens per launch, i.e. the prefill chunk. auto sizes it to a fifth of the memory left after the state pool. |
--num-spec |
Speculative depth. The optimum is workload-dependent; 2 is a robust default. |
--kv-cache-dtype |
bf16 or int8. int8 halves KV bytes: more decode throughput at long context, less prefill. |
--limit-mm-per-prompt |
Images one request may carry. |
Roadmap
- More models
- Tensor parallelism at any degree
- NPU/GPU co-working prefill
- Native INT4 quantization
License
- SnowLLM: Apache-2.0
- SnowLLM-Kernels: All rights reserved
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file snowllm-0.1.2-py3-none-any.whl.
File metadata
- Download URL: snowllm-0.1.2-py3-none-any.whl
- Upload date:
- Size: 79.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
86cfe7df92efcfffaf6fc20341618fb7d1af6f810df90875a5af6aa95fbcebe4
|
|
| MD5 |
d74113594f59d49428b5ede5746f694c
|
|
| BLAKE2b-256 |
f1395e404216d65c203c8099428bfbe55001bf0a753282b11ac50fdf4861a87d
|
Provenance
The following attestation bundles were made for snowllm-0.1.2-py3-none-any.whl:
Publisher:
pypi.yml on SnowLLM/SnowLLM
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
snowllm-0.1.2-py3-none-any.whl -
Subject digest:
86cfe7df92efcfffaf6fc20341618fb7d1af6f810df90875a5af6aa95fbcebe4 - Sigstore transparency entry: 2369563395
- Sigstore integration time:
-
Permalink:
SnowLLM/SnowLLM@d70d69c8d70136111b1dbaade20af03eef9d3d5c -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/SnowLLM
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
pypi.yml@d70d69c8d70136111b1dbaade20af03eef9d3d5c -
Trigger Event:
push
-
Statement type:
