한국어 | English

FiTuna

Stop guessing your llama.cpp config. Measure it.

Give it a model file, a target speed (tok/s) and a quality budget (%), and it finds the llama.cpp configuration — quantization level, GPU offload and context length — that actually hits those numbers on your machine: the highest-quality quant that reaches your speed, with the fewest GPU layers it needs, proven by real benchmarks.

pip install fituna

▶ Demo video · Korean reproduction guide for judges

Why

Running a local LLM through llama.cpp means making three choices: how hard to compress the model (quantization level, Q2–Q8 — lower is faster but loses quality), how many layers to put on the GPU (-ngl), and how long a context to allow. That's dozens of combinations, and today most people search them by trial and error.

What you actually want answered are three questions. ① Will this machine hit my target speed? ② What is the quality loss, in percent? ③ Which config keeps the most quality while still meeting the target? — and no tool answered them:

Runners like Ollama and LM Studio are absent from this table on purpose — they execute a given config rather than answer these questions, a different and complementary job. The config FiTuna finds can be handed straight to Ollama via --export-ollama.

And once you actually measure, intuition keeps getting overturned: on an Apple M3 Pro the "obviously best" Q8_0 failed the speed target, and the answer wasn't a quant alone but Q4_K_M plus the minimal offload -ngl 33. No amount of spec-sheet reading produces that answer.

Try it in 2 minutes

The fastest way to see it work for yourself: a full pipeline run (download → quantize → quality-gate → benchmark) on a small 258 MB model finishes in about a minute on an M-series Mac. See the full flow first in the demo video:

brew install llama.cpp python@3.13
python3.13 -m venv .venv
source .venv/bin/activate
pip install fituna
fituna fetch-corpus --lang en --out wiki.txt
fituna run --hf bartowski/SmolLM2-135M-Instruct-GGUF \
  --target-tps 240 \
  --max-quality-loss 5 \
  --ctx 2048 \
  --quality-corpus wiki.txt \
  --out ./out --resume

What each fituna run flag means:

¹ Already have a model file on disk? Use --model <path.gguf> instead of --hf.

Example result

When the command finishes, you get output like this (a real run on an Apple M3 Pro):

FiTuna result: MEETS TARGET

  quant           : Q8_0        # <- the optimal quantization level it found
  ngl             : 26          # <- the minimum GPU offload layers meeting the target
  ctx             : 2048        # <- the verified context length

  prompt tok/s (pp): 2017.64
  gen tok/s    (tg): 261.78     # <- measured generation speed -- clears the 240 target

  perplexity      : 18.2931 (baseline 18.2407)
  quality loss    : 0.29%       # <- measured quality loss -- within the 5% budget

  artifact: out/SmolLM2-135M-Instruct-8078a5b74b5a-Q8_0.gguf  (144.8 MB -- already produced during the search)

  1) local API server (OpenAI-compatible):
       /opt/homebrew/bin/llama-server -m out/SmolLM2-135M-Instruct-8078a5b74b5a-Q8_0.gguf -ngl 26 -c 2048 --port 8080
  2) import into Ollama: re-run with --export-ollama to write a Modelfile beside the artifact
  3) terminal chat (interactive check):
       /opt/homebrew/bin/llama-cli -m out/SmolLM2-135M-Instruct-8078a5b74b5a-Q8_0.gguf -ngl 26 -c 2048

How to read it: the first line is the verdict — MEETS TARGET means a configuration satisfying your target was found, followed by that configuration (quant × ngl × ctx) and the measured evidence (speed and quality loss). The quantized model on the artifact: line was already produced during the search. Re-running the same command with --resume reuses its stored measurements. Absolute numbers and the winning quant can vary by machine and by run — measured run-to-run variance.

Measured results

Three models of different sizes, each given a target. In all three runs the "safe default" Q8_0 failed the speed target:

Apple M3 Pro, llama.cpp build 9960. Full logs and run-to-run variance: docs/RESULTS.md · Scenarios: docs/USE_CASES.md · Reproduce on NVIDIA/Linux (free T4):

Couldn't you just ask a chatbot?

We tested that question under the same conditions used in the result report. FiTuna and three chatbots were given Qwen3-4B, a 30 tok/s speed target, and a 5% quality-loss budget. Each proposed configuration was then run three times on the same Apple M3 Pro.

¹ Mean ± between-run standard deviation over three independent runs, with five internal measurements per run. Quality is the perplexity increase from F16 on 32 WikiText-2 chunks.

All three configurations that could be measured stayed within the quality budget. Only FiTuna met the speed target on every run, while using three fewer GPU-offloaded layers than Claude's proposal. ChatGPT proposed the smallest offload but passed only once. This is not a general chatbot ranking; it shows that a minimal passing configuration for one machine needs repeated measurement. Full conditions, per-run values, and limitations: docs/CHATBOT_COMPARISON.md

How it works

flowchart LR
    A["Input<br/>F16 GGUF<br/>target tok/s · quality budget"] --> B["Stage 1 · measure quality<br/>quantize every candidate,<br/>measure perplexity"]
    B --> C{"quality<br/>gate"}
    C -->|"fail (early-exit A)"| X["next candidate"]
    C -->|"pass · sort by measured quality"| D["Stage 2 · measure speed<br/>llama-bench"]
    D --> E{"target met at<br/>full offload?"}
    E -->|"miss (early-exit B)"| X
    E -->|met| F["binary-search ngl<br/>minimal offload"]
    F --> G["Artifacts<br/>quantized .gguf +<br/>llama-server command"]
    B -.store measurements.-> H[("sqlite3 cache<br/>--resume < 1 s")]
    D -.-> H

FiTuna works in two stages. Stage 1 quantizes every candidate and measures its quality loss first — you can't rank candidates by a number you haven't measured. Stage 2 then benchmarks them in that measured quality order, dropping any quant that misses the target without wasting further benches. Every measurement lands in an sqlite3 cache whose key includes the llama.cpp build version, so upgrading the engine never silently reuses stale numbers.

See how it works in detail: docs/ARCHITECTURE.md

Install

python3.13 -m venv .venv
source .venv/bin/activate
pip install fituna

Python 3.11+ is required. Any installed 3.11+ interpreter works — swap the first line accordingly (e.g. python3.12). The macOS system python3 (3.9.6) is too old; if you have no 3.11+ at all, brew install python@3.13. Zero runtime dependencies. You also need llama.cpp:

brew install llama.cpp        # macOS/Linux Homebrew

# llama.cpp from source (any platform; add -DGGML_CUDA=ON for NVIDIA)
git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B llama.cpp/build && cmake --build llama.cpp/build --config Release
# then add --llama-bin-dir llama.cpp/build/bin to fituna commands

# FiTuna for development
git clone https://github.com/leeyunseokarchive/fituna
python3.13 -m venv .venv && source .venv/bin/activate
pip install -e fituna

Commands

New here? fituna quickstart is the easiest entry — a wizard walks you from environment check to a finished search without knowing any of the commands below. Full options for each: fituna <command> -h.

fituna fetch-corpus --lang en --out wikitext-2-raw-test.txt        # wikitext-2
fituna fetch-corpus --lang ko --out kowiki-corpus.txt --rows 500   # Korean Wikipedia

from fituna.hardware import detect_hardware

hw = detect_hardware()
print(f"{hw.gpu_vendor.value}: {hw.gpu_name}, {hw.vram_mb} MB VRAM")
# apple: Apple M3 Pro, 18432 MB VRAM

MCP server — measured answers for AI agents

Ask a chatbot "which local model config fits my machine?" and you get a guess derived from spec sheets. Connect FiTuna's MCP server and the agent gets measured results instead:

claude mcp add fituna -- fituna-mcp      # any MCP client with stdio transport

Stdlib-only JSON-RPC 2.0 over stdio, no SDK (mcp_server.py).

Scope and limitations

FiTuna recommends, and stops there. The output is the quantized .gguf already produced during the search, plus llama-server/llama-cli commands you copy and run (and an Ollama Modelfile with --export-ollama) — actually serving the model stays llama.cpp's job (rationale). Current limitations:

• Results are valid only on the machine that ran them — FiTuna never extrapolates another machine's numbers from a spec sheet. Need a config for a different machine? Run FiTuna there (it's a cross-platform CLI). Machines disagreeing is exactly why measurement beats estimation — same model, opposite verdicts on M3 Pro vs T4
• Single GPU only — no --tensor-split
• No Windows AMD auto-detection — pass --gpu amd --vram-mb <N>
• Quality = perplexity on the corpus you choose — a proxy; measure on text resembling your workload. --quality-metric kld adds KL divergence to the report, but the pass/fail gate is still perplexity (#49)
• Verdicts depend on --ppl-chunks — re-measure candidates close to your budget (measured effect)
• Benchmarks are thermally sensitive — verdicts within a few tok/s of target are marginal (variance analysis)
• Real-hardware E2E covers macOS · Linux — Windows is unit-tested and CI-run

Roadmap

Ordered by how directly each item narrows the limitations above. Each item is tracked on GitHub Issues.

• KLD as the quality gate — --quality-metric kld now reports KL divergence; next, let it decide pass/fail (#49)
• llama.cpp --fit integration — use upstream's memory auto-fit as a sub-routine to seed the ngl search
• Distributed measurement — quantize and score quality on a remote server, then measure only speed on resource-constrained target devices
• Edge devices (Raspberry Pi, Jetson) — run only llama-bench on the device, one candidate at a time, so a board too small for the full search can still be measured (#56)
• --launch and LM Studio preset export — additional output formats, keeping the same artifact boundary as --export-ollama
• Multi-GPU (--tensor-split) search — lift the single-GPU limitation

Contributing

The codebase is small, dependency-free and contract-first — start at fituna/config.py. The full unit suite and a 3-OS × 2-Python CI matrix guard it. See CONTRIBUTING.md and docs/DEVELOPMENT.md.

License

MIT © FiTuna contributors · THIRD_PARTY_NOTICES.md · SBOM · Open-source usage · AI-assisted development disclosure · CHANGELOG.md · SECURITY.md