clanker-gguf 1.0.1

LLM Turbo-Optimizer CLI. detect hardware, parse GGUF models, generate optimized llama-server commands.

clanker

generates llama-server commands so you don't have to guess how many layers fit on your gpu. reads the gguf header, does the math, prints the command.

i got tired of trial-and-error vram tuning for moe models on consumer gpus.

Setup

requires python 3.10+ and pip.

pip install clanker-gguf

add extras as needed:

pip install "clanker-gguf[gguf]"     # gguf header parsing
pip install "clanker-gguf[hf]"       # huggingface download support
pip install "clanker-gguf[nvidia]"   # nvidia-ml-py fallback
pip install "clanker-gguf[all]"      # everything above

or install from source:

pip install git+https://github.com/Stavros-alt/clanker.git
pip install "git+https://github.com/Stavros-alt/clanker.git#egg=clanker-gguf[all]"

or clone and install locally:

git clone https://github.com/Stavros-alt/clanker.git
cd clanker
pip install -e ".[all]"

you also need:

• nvidia-smi on PATH (cpu-only mode works without it)
• llama-server from ik_llama.cpp (build it with clanker build, or install manually)
• windows? good luck. this probably works on wsl.

Usage

clanker run <model>

clanker run ~/models/qwen.gguf          # direct path
clanker run qwen                         # fuzzy cache search
clanker run qwen.gguf --preset speed     # apply a preset
clanker run qwen.gguf --context 65536    # override context
clanker run qwen.gguf --execute          # run it

defaults to 8K context, q8_0 KV cache, no flash attention. pass --preset big-brain for the optimized 128k config.

clanker download <repo>

clanker download unsloth/Qwen3.6-35B-A3B-GGUF            # scan only
clanker download unsloth/Qwen3.6-35B-A3B-GGUF --execute   # download
clanker download unsloth/Qwen3.6-35B-A3B-GGUF -c 32768    # budget for 32k ctx

picks the highest-quality quant that fits your ram, with preference for models whose backbone fits on gpu. --fit handles the offloading at runtime.

Other commands

clanker discover         # scan hf cache for gguf models
clanker discover --json  # json output
clanker ls               # alias for discover
clanker search           # open hf search with hardware-tuned bounds
clanker presets          # list all presets with their settings
clanker build            # clone and compile ik_llama.cpp

Presets

all presets enable mtp speculative decoding when the model supports it.

Name	Context	KV Cache	Notes
big-brain	128K	k=q8_0, v=q5_1	mlock, no-mmap, flash-attn
speed	32K	k=q6_0, v=q5_0	mlock, flash-attn
infinite	512K	k=q5_0, v=q4_1	no-mmap, flash-attn
coding	64K	k=q8_0, v=q5_1	mlock, flash-attn, temp=0.2

kv cache quants are from KLD benchmarks (anbeeld). mixed k/v pairs outperform symmetric ones on the pareto frontier.

How it works

1. reads your gpu vram via nvidia-smi
2. detects physical cpu cores (uses cores - 1 for thread count)
3. parses the gguf header for architecture, layers, experts, quant type
4. figures out layer split, kv cache size, fit-margin based on available vram
5. spits out a llama-server command with thread tuning, ubatch tuning, and mlock

Notes

• --fit --fit-margin N means "keep N MB of vram free". base is 1664 for ik_llama, 4608 for mainline. mtp adds 2048 mb overhead.
• thread count is physical_cores - 1 (min 1). keeps one core free for gpu scheduling and os tasks.
• -ub 2048 sets the micro-batch size for max memory-bandwidth utilization during prompt prefill.
• mtp flags are only added when the model file has "mtp" in its name.
• the downloader tries llama-cli -hf first, falls back to huggingface_hub. if llama-cli downloads then ooms, clanker detects the existing file and skips re-download.
• quantization detection from gguf metadata is unreliable (returns int enums). filenames are more accurate, so clanker parses the filename.
• combined expert tensors (qwen3, deepseek) are handled by dividing total size by expert_count from metadata.
• --backend llama switches to mainline llama.cpp flags (spec-type, spec-draft-n-max, etc). default is ik_llama.