ggufit

Check whether a local LLM can run on your machine, CPU-only — from PowerShell or any terminal.

Uses real formulas (model memory footprint, KV cache size, and a measured memory-bandwidth micro-benchmark) rather than guesses, to estimate whether a model fits in RAM and roughly how fast it'll generate tokens. Correctly handles MoE models (speed driven by active experts, not total params) and SSM/MLA architectures (Mamba, DeepSeek's MLA) that don't use standard multi-head attention.

Install

# Recommended - works everywhere, gives you a global `ggufit` command
pipx install ggufit

# Or plain pip (Windows PowerShell, or inside a venv on Linux/macOS)
pip install ggufit

Installing from source (for development)

git clone <this repo>   # or unzip the source
cd ggufit/               # the folder containing pyproject.toml
pipx install -e .        # editable install - code changes apply without reinstalling
# or: pip install -e .   (inside a venv, or with --break-system-packages)

Usage

# Full hardware scan: shows your CPU/RAM/bandwidth and which models fit
ggufit scan

# Check one specific model
ggufit llama3.1
ggufit mistral
ggufit qwen2.5-14b

# Force a specific quantization level
ggufit llama3.1 --quant q4

# Evaluate at a longer context length
ggufit qwen2.5-32b --context 16384

How it works

For each model:

1. Model size (RAM) = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
2. KV cache = 2 × layers × hidden_size × context_len × bytes_per_param × kv_cache_multiplier
   • kv_cache_multiplier defaults to 1.0 (standard MHA), and is set lower for architectures that don't use full attention at every layer: 0.0 for pure SSM/Mamba (no attention at all), ~0.15 for MLA (DeepSeek-V2/V3/R1, MiniCPM3), ~0.125 for hybrid Mamba+attention (Jamba).
3. Total RAM needed = (model size + KV cache) × 1.2 overhead factor
4. Speed estimate = measured_memory_bandwidth / active_size, where active_size is the total model size for dense models, or just the active experts' size for MoE models (active_params_billion) — since only those weights are streamed from RAM per token.

ggufit runs a quick real memory-bandwidth benchmark (a large array copy) instead of guessing from RAM specs, since achievable bandwidth depends heavily on channel configuration.

Adding models

Edit ggufit/models_db.py and add an entry to the MODELS dict with params_billion, layers, and hidden_size (found in the model's Hugging Face config.json). Optional:

• moe: True + active_params_billion: X for Mixture-of-Experts models
• kv_cache_multiplier: X for non-standard attention architectures (see above)

Known limitations

• Speed estimates assume batch size 1, single-user chat.
• The bandwidth benchmark is single-threaded; real inference engines use multiple threads.
• Quant byte-per-param values are approximations of real GGUF file sizes.
• KV cache still assumes full hidden_size for standard (non-flagged) models even though most modern ones use GQA with fewer KV heads than query heads — this is intentionally conservative (over-estimates KV, doesn't affect the dominant "does it load" answer).
• kv_cache_multiplier values for MLA/hybrid architectures are approximate, not per-model-measured.

Notes

This is a side-project CLI. The full hardware-scan desktop app (Rust/Tauri) is a separate, more thorough tool still in development.