---

ggufit sizes local LLMs against your hardware: it measures your real memory bandwidth, computes each model's true memory footprint from its actual architecture, and tells you what fits and roughly how fast it will generate.

It is CPU-only by design, has zero dependencies, and makes no network calls — the model catalogue ships with it.

pipx install ggufit     # recommended: isolated env, global `ggufit` command
pip install ggufit      # or inside a venv

See it work

$ ggufit scan --for code --fastest --top 4

=== Hardware Profile ===
  OS:              Linux 6.2.0-39-generic
  Logical cores:   8
  RAM total:       7.44 GB
  RAM free now:    1.96 GB
  Measured bandwidth: 17.5 GB/s (multi-process, real)

=== Models For This Machine (code) ===
  With RAM cleared: 5.44 GB usable (everything but 2 GB reserved for the OS)
  Right now: 1.96 GB free, with what you have open left alone
  Sorted by estimated speed, quickest first.

  Runs right now (2):
    - Qwen2.5-Coder 0.5B  [Q8]  ~21.6-40.0 tok/s
    - Qwen2.5-Coder 1.5B  [Q3]  ~14.5-26.9 tok/s

  Usable if you free up RAM (12):
    - CodeGemma 2B  [Q8]  ~4.2-7.9 tok/s
    - DeepSeek-Coder 1.3B  [FP16]  ~4.1-7.7 tok/s
      ... and 10 more (raise --top to see them)

  Will not fit on this machine (17):
    - Code Llama 13B
    - Code Llama 34B
      ... and 15 more (raise --top to see them)

Or drill into one model:

$ ggufit llama3.1

=== Llama 3.1 8B @ 4096 ctx ===
  Right now:        does not fit
      Needs 4.72 GB at Q2, only 2.01 GB is free.

  With RAM cleared: runs at Q2   <- free up RAM and this becomes usable
      Model size: 2.99 GB
      KV cache:   0.5 GB
      Total need: 4.72 GB of 5.44 GB
      Speed:      ~3.6-6.8 tok/s  (memory-bandwidth bound, uncalibrated +/-30% range)
      Note: Q3 would need 5.52 GB, short by 0.08 GB.

Why not just multiply parameters by bytes?

Because that gets three things wrong, and each one is worth gigabytes:

	The naive answer	What ggufit does
Attention cache	assume every model caches per attention head	reads each model's real K/V head counts — a 4-8x difference on modern models
Mixture-of-experts	score a 671B model as if all 671B are read per token	fits on total weights, but times on active experts only
Memory bandwidth	look up the RAM spec sheet	measures it, multi-process, on your machine

Concretely: Llama 2 70B at 32K context needs 10 GB of attention cache, not the 80 GB a per-head assumption gives you. That is the difference between "you need a server" and "this runs on a workstation."

Mamba and DeepSeek's MLA are handled as themselves too — a pure state-space model has no attention cache at any context length, and MLA caches one compressed vector per layer.

Two answers, not one

Every result is reported against two memory figures, because "can this machine run it" and "can it run it without closing anything first" are different questions, and one number quietly conflated them:

• Right now — what is actually free at this instant, with everything you have open left alone. A live reading, so it moves between runs.
• With RAM cleared — total installed RAM minus a reserve for the OS (2 GB or 15%, whichever is larger). Fixed and reproducible: the machine's real ceiling.

That is why scan has a middle group. Usable if you free up RAM means the model fits your hardware fine — just not alongside your browser. Where free memory cannot be read, the two groups collapse into one rather than guessing.

Usage

# What runs on this machine
ggufit scan

# Filter by what you want the model for
ggufit scan --for code          # also: reasoning, math, vision, chat, general

# Sort by speed, cap the output
ggufit scan --fastest
ggufit scan --for code --fastest --top 5

# Drill into one model (short names, HF-style and Ollama-style all work)
ggufit llama3.1
ggufit llama-3.1-8b
ggufit llama3.1:8b

# Force a quant level, or test a longer context
ggufit llama3.1 --quant q4
ggufit qwen2.5-32b --context 16384

Output is plain text when piped (ggufit scan > report.txt), coloured when it is a terminal. NO_COLOR and FORCE_COLOR are both honoured.

Installing from source

cd ggufit/                # the folder containing pyproject.toml
pipx install -e .         # editable install
# or: pip install -e .    (inside a venv, or with --break-system-packages)

How it works

For each model:

1. Model size (RAM) = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
2. KV cache uses the model's real attention architecture, not a flat guess:
   • Standard / GQA models: 2 × layers × num_kv_heads × head_dim × context_len × bytes_per_elem, using each model's actual KV head count (sourced from its Hugging Face config.json), not hidden_size. Most models since Llama 2 70B use Grouped-Query Attention, where num_kv_heads is far smaller than the number of query heads — using hidden_size there over-estimates the KV cache by 4-8x (e.g. Llama 2 70B at 32K context: 10 GB real vs 80 GB under the old formula).
   • MLA (DeepSeek-V2/V3/R1, MiniCPM3): layers × (kv_lora_rank + qk_rope_head_dim) × context_len × bytes_per_elem — MLA caches one compressed latent vector per layer, not one per head.
   • Pure SSM/Mamba (no attention at all): 0 — a fixed-size recurrent state that doesn't grow with context length.
   • A handful of older/override-only models have no published head-count data; those still fall back to the conservative hidden_size-based estimate.
3. Total RAM needed = weights × overhead_multiplier(weights) + KV cache
   • The overhead multiplier accounts for compute buffers, activations, and allocator overhead. It is size-dependent, not a flat percentage: a fixed ~1GB baseline buffer plus ~7.6% of weight size. This means it's high for tiny models (~2x for a 1GB model, which genuinely needs ~1GB of buffers on top) but asymptotes to ~1.08x for very large models. Calibrated against real-world llama.cpp memory reports across 7B/13B/70B/671B models — e.g. DeepSeek-V3 671B at Q4 comes out to ~405GB, matching reality, instead of the ~450GB an old flat-20% rule would have predicted.
4. RAM budget — two of them, see "Two answers, not one" above. The reproducible one is total installed RAM minus max(2 GB, 15% of total) reserved for the OS; the live one is whatever is free at this instant. The reproducible figure is what the fit verdict is anchored to, since a live reading alone used to make ggufit report a different verdict for the same machine seconds apart.
5. Speed estimate = measured_memory_bandwidth / active_size, shown as a range (±30%, since the runtime-efficiency constant behind it is not yet calibrated against real measurements — see Known limitations). active_size is the total model size for dense models, or just the active experts' size for MoE models (active_params_billion) — since only those weights are streamed from RAM per token.

ggufit runs a real memory-bandwidth benchmark (several worker processes copying a buffer concurrently, STREAM-style) instead of guessing from RAM specs, since achievable bandwidth depends heavily on channel configuration. It has no dependencies — standard library only.

Model catalogue

The 218-model catalogue (ggufit/models.json) is generated from HuggingFace config.json / safetensors metadata, not hand-typed — see the ggufit repo's tools/fetch_model_specs.py. This is what makes the per-model KV head counts above trustworthy: a hand-maintained file couldn't carry that data without it silently going stale. A few fields (MoE active_params_billion, and specs for repos HuggingFace can't resolve directly) come from a small manual overrides file instead, since those genuinely aren't derivable from a config.

Known limitations

• Speed estimates assume batch size 1, single-user chat, and are shown as an uncalibrated ±30% range — the runtime-efficiency constant behind them is a placeholder pending real tokens/sec measurements.
• Quant byte-per-param values are approximations of real GGUF file sizes.
• A handful of older or override-only models have no published attention head-count data and fall back to the conservative hidden_size-based KV estimate (over-estimates, never under-estimates).
• GPU/VRAM is not modeled at all — every figure is CPU-only. For a machine with a GPU, these numbers don't reflect what you'd actually get running on it.
• The overhead multiplier is calibrated against a handful of real-world data points (7B/13B/70B/671B at Q4); it's a strong approximation but not exact for every model/runtime.
• Prompt processing time isn't modeled — only generation speed.

Changelog

Upgrading from 0.1.2? UPDATE.md, shipped in the source distribution, covers what changed and why, the measured before/after figures, and the library-API breaking changes.

• 0.1.3 — KV cache now uses each model's real attention head counts (GQA) or MLA-specific fields instead of a flat hidden_size estimate, fixing a 4-8x over-estimation on most modern models (e.g. Llama 2 70B at 32K context: 80 GB → 10 GB, the correct figure). The memory-bandwidth benchmark is now a real multi-process STREAM-style measurement instead of a single-threaded numpy copy that read a fraction of true bandwidth. Results are now reported against two RAM figures — what's free right now and the machine's ceiling with RAM cleared — with a new Usable if you free up RAM group in scan for models that fit the hardware but not the current moment. New --for <use> filter (code, reasoning, math, vision, chat, general), --fastest sort and --top N limit. Speed is now shown as a range, not a false-precision single number. Dropped the psutil/numpy dependencies — the CLI is standard-library only now. Model catalogue expanded from 217 hand-typed entries to 218 generated from HuggingFace.
• 0.1.2 — Overhead is now size-dependent (fixed ~1GB baseline + ~7.6% of weights), calibrated against real llama.cpp memory reports. Fixes large-model RAM over-estimation (DeepSeek-V3 671B Q4 now estimates ~405GB, matching reality, vs ~450GB before). Also fixes a wording bug where the "try a lower quant" hint could suggest the same quant that just failed.
• 0.1.1 — Custom use-only license; corrected install instructions.
• 0.1.0 — Initial release (MoE active-param speed, SSM/MLA KV cache handling, 217 models).

License

ggufit uses a custom use-only license, not an open-source one. You may install and run it for any purpose including commercial use, and redistribute unmodified copies. You may not modify it, redistribute modified versions, or sublicense it. Full terms in the LICENSE file.

Version 0.1.0 was published under MIT; that release stays MIT for anyone who already has it. License changes apply going forward, not retroactively.

Q&A

A comprehensive FAQ — from "what is a GGUF" to the exact formulas behind every number ggufit prints. Organized beginner → expert.

Basics

What is a GGUF? GGUF (GPT-Generated Unified Format) is a file format for storing LLM weights, designed by the llama.cpp project for fast loading and CPU/GPU-flexible inference. It bundles the model's tensors and metadata (architecture, tokenizer, etc.) into a single file. It replaced the older GGML format. If you've downloaded a .gguf file from Hugging Face to run in llama.cpp, Ollama, or LM Studio, that's what ggufit is estimating compatibility for.

What is quantization? Shrinking a model's weights from their original precision (usually 16-bit floats) down to smaller representations (8-bit, 4-bit, etc.) to save memory and speed up inference, at the cost of some accuracy. A "Q4" model uses roughly a quarter of the memory of the same model at full 16-bit precision.

What do Q4, Q5, Q6, Q8 mean? The number is roughly the average bits per weight after quantization (not exactly, see next question). Lower number = smaller file, faster inference, more quality loss. Q4 is the most common "sweet spot" for CPU-only local inference. Q8 is close to lossless but nearly as large as full precision.

Why isn't Q4 exactly 4 bits then? Modern GGUF "K-quants" (the _K_M, _K_S suffixes you see on Hugging Face) don't use a uniform bit-width across the whole model — they mix precision per tensor, using slightly higher precision for the parts most sensitive to quality loss. So "Q4" is really an average around 4-5 bits/weight in practice. ggufit uses effective per-quant byte values that reflect this real-world average, not naive N-bit math:

Quant	Effective bytes/param
FP32	4.0
FP16/BF16	2.0
Q8	1.05
Q6	0.8
Q5	0.7
Q4	0.6
Q3	0.5
Q2	0.4

What's the difference between RAM and VRAM, and why does ggufit only care about RAM? VRAM is memory on a dedicated GPU; RAM is your system's main memory, used by the CPU. ggufit is CPU-only by design — it answers "can my CPU and system RAM handle this," not "can my GPU handle this." If you have a GPU, tools like nvidia-smi and the model card's VRAM requirements are what you want instead.

What is context length? The number of tokens (roughly, chunks of a word) the model can "see" at once — your prompt plus its response so far. Longer context means the model can process longer documents, conversations, and code, but it also means more memory used for the KV cache (see below).

What are tokens/sec, and what's a "good" number? How many tokens the model generates per second. For a comfortable reading pace, most people find 5-15 tok/s tolerable for chat; below ~2 tok/s feels quite slow; above 20 tok/s feels close to instant. It's highly subjective and task-dependent though — background batch jobs can tolerate much lower throughput than an interactive chat.

---

Installing and running ggufit

How do I install it?

pipx install ggufit    # recommended
pip install ggufit     # inside a venv, or with --break-system-packages

Why do I get "externally-managed-environment"? Modern Debian/Ubuntu (PEP 668) blocks system-wide pip install to protect the OS's own Python packages. Use pipx instead — it installs into an isolated environment while still giving you a global command.

Why does ggufit say "command not found" right after installing? Either your venv isn't activated (source venv/bin/activate), or pipx's bin directory isn't on your PATH yet (run pipx ensurepath and reopen your terminal).

Why did a model that should fit show as "does not fit"? Check which of the two verdicts you're reading. "Right now" reflects what's free at this instant, so other programs holding memory will fail it; "with RAM cleared" is the machine's real ceiling. A model in the Usable if you free up RAM group fits your hardware fine — it just doesn't fit alongside what you currently have open.

Can I check a model at a specific quant instead of letting ggufit auto-pick? Yes — ggufit <model> --quant q4. Without --quant, it auto-picks the highest-quality quant that fits.

How do I find models for a specific job, or just the fastest ones? ggufit scan --for code filters to coding models; --for also takes reasoning, math, vision, chat and general. Add --fastest to sort by estimated speed and --top N to cap the list, e.g. ggufit scan --for code --fastest --top 5. Tags are editorial groupings held in tools/aliases.json, not something the model files declare, so treat them as a helpful index rather than an authoritative capability list — plenty of "general" models write perfectly good code.

---

Understanding the numbers (intermediate)

How is "Model size (RAM)" calculated?

model_size_bytes = num_parameters × bytes_per_param

bytes_per_param comes from the quant table above. This is the number of bytes the weights occupy once loaded — the dominant factor in "will this even load."

What is the KV cache, and why does it matter? During generation, the model caches the Key and Value tensors from every previous token in the conversation so it doesn't have to recompute them each step. This cache grows with context length. The standard formula:

kv_cache_bytes = 2 × num_layers × hidden_size × seq_len × batch_size × bytes_per_param

The 2× covers storing both K and V. This is why a model that fits fine at a 4K context can stop fitting at 32K — the KV cache scales linearly with context length while the model weights stay fixed.

Why is there an "overhead factor" on top of model size + KV cache? Real inference isn't just raw tensor storage — the runtime, OS, and memory allocator all need working space too (buffers, fragmentation, temporary activations). It's a fixed ~1GB baseline plus ~7.6% of weight size (not a flat multiplier — see "How it works" above), calibrated against real llama.cpp memory reports.

Why is CPU inference speed based on memory bandwidth instead of raw compute (FLOPS)? At batch size 1 (the normal case for a single person chatting), generating each new token requires reading every single model weight from RAM once. The CPU spends far more time waiting on memory than doing arithmetic — so the bottleneck is how fast data can move from RAM to the CPU, not how many operations per second the CPU can do. That's why the formula is:

tokens_per_sec ≈ memory_bandwidth (GB/s) / model_size (GB)

Why does ggufit benchmark my memory bandwidth instead of just knowing it from my RAM specs? Achievable bandwidth depends on RAM generation, channel configuration (single/dual/quad-channel), and platform quirks — none of which are reliably detectable across Windows/Mac/Linux without vendor-specific tools. So ggufit runs several worker processes copying a buffer concurrently on your actual machine, right now (a single thread can't saturate a memory controller multiple cores can drive), and uses that measured number instead of guessing.

Why does the bandwidth number change slightly every time I run ggufit? It's a live micro-benchmark, not a cached constant — normal system load, thermal throttling, and other processes competing for memory access all cause small run-to-run variance. That's expected and not a bug.

---

Architecture-specific math (expert / advanced)

What is a Mixture-of-Experts (MoE) model, and why does it need special handling? An MoE model has many "expert" sub-networks, but only a subset of them are activated for any given token (a small router network decides which experts to use). This means:

• All experts must be loaded into RAM — because any token could route to any expert, so the fit check uses the full params_billion (total, all experts).
• Only the active experts are actually read from RAM per token — so the speed estimate uses active_params_billion instead. Using total params for speed would make big MoE models look absurdly slow (DeepSeek-V3 at 671B total but only 37B active would look ~18x slower than it really is if you used the total).

fit check:   uses params_billion (total) — all experts must be resident
speed check: uses active_params_billion — only active experts are read per token

What is GQA (Grouped-Query Attention), and how does ggufit account for it? Most modern transformer models don't give every attention head its own K/V projection — they group multiple query heads to share a smaller number of K/V heads (num_kv_heads < num_attention_heads). This makes the real KV cache smaller than a formula assuming hidden_size worth of K/V per layer would predict — over-estimating by 4-8x for models like Llama 2 70B or Mistral 7B. ggufit uses each model's actual num_key_value_heads and head_dim (sourced from its Hugging Face config.json) to compute the real KV cache size, not a blanket correction factor — that matters because some models (Phi-3-mini) are genuinely full multi-head attention, and a global multiplier would break those while fixing everything else. Only a handful of older or override-only models still fall back to the conservative hidden_size estimate, because no published head-count data exists for them.

What is MLA (Multi-head Latent Attention)? MLA (used by DeepSeek-V2/V3/R1 and MiniCPM3) compresses the K/V representations into a much smaller latent vector before caching them, then reconstructs full K/V on the fly. ggufit computes this directly from the model's real kv_lora_rank and qk_rope_head_dim fields — one compressed latent vector cached per layer, not one per head — rather than applying an approximate correction factor.

Why do pure Mamba/SSM models show a 0 GB KV cache? State-Space Models (Mamba, used in Codestral Mamba and Falcon-Mamba) don't use attention at all — they maintain a fixed-size recurrent state instead of caching every previous token's K/V. That state doesn't grow with context length. So their KV cache is genuinely zero, regardless of how long the context gets. This is why ggufit falcon-mamba --context 65536 still shows ~0GB KV cache even at a huge context length.

What about hybrid models like Jamba? Jamba interleaves Mamba blocks with regular attention blocks — only a fraction of layers actually use attention. ggufit doesn't yet have published per-layer attention-head data for these hybrid architectures, so they fall back to the conservative hidden_size-based estimate rather than a wrong precise-looking number.

Why does ggufit need real layers and hidden_size values instead of estimating them from param count? Because two models with the same parameter count can have very different KV cache sizes depending on how those parameters are distributed across layers and hidden dimension — there's no reliable shortcut from param count alone. ggufit stores these values explicitly per model (sourced from each model's Hugging Face config.json) to keep the KV cache estimate accurate.

---

Project & contributing

How do I add a model that isn't in the database? The catalogue (ggufit/models.json) is generated, not hand-edited — see the ggufit repo's tools/fetch_model_specs.py. Add the model's HuggingFace repo id to tools/repos.txt, re-run the fetcher, and (for MoE models, or repos HuggingFace can't resolve directly) add an entry to data/overrides.json for whatever the fetcher couldn't derive on its own — most commonly active_params_billion.

Why was the project renamed from moscan to ggufit? The name moscan was already taken on PyPI. ggufit was chosen instead — pun on GGUF (the file format) + "fit" (does the model fit on your machine).

What license is ggufit under? A custom "use-only" license: you're free to install and run it for any purpose, including commercial use, but you may not modify it or redistribute a modified version. See the LICENSE file for exact terms. Note: versions published before this license was adopted (0.1.0) remain under their original MIT terms for anyone who already obtained that specific release — license changes only apply going forward, not retroactively.

Is this a substitute for actually running the model to see how it performs? No — treat every number here as an estimate to guide a decision (e.g. "should I even attempt downloading this 40GB file"), not a guarantee. Real-world speed depends on your specific inference engine (llama.cpp, Ollama, etc.), thread count settings, OS scheduler behavior, and quantization implementation quality, none of which ggufit can measure without you actually running the model.