$ fituna run --model Qwen3-4B-Instruct-2507-F16.gguf \
    --target-tps 30 --max-quality-loss 5 --ctx 4096 --wikitext wiki.txt --out ./out

[Q6_K]   full-offload 28.48 tok/s < target 30.00, skipping (early-exit B)
[Q8_0]   full-offload 24.22 tok/s < target 30.00, skipping (early-exit B)
[Q5_K_M] full-offload 29.59 tok/s < target 30.00, skipping (early-exit B)
[Q4_K_M] found ngl=33 meeting target -- done

FiTuna result: MEETS TARGET
  quant : Q4_K_M   ngl : 33   ctx : 4096
  gen tok/s : 30.81      quality loss : 1.73%

  artifact: out/Qwen3-4B-Instruct-2507-...-Q4_K_M.gguf  (2.3 GB -- already produced during the search)

  1) local API server (OpenAI-compatible):
       llama-server -m out/Qwen3-...-Q4_K_M.gguf -ngl 33 -c 4096 --port 8080
  2) import into Ollama: re-run with --export-ollama to write a Modelfile beside the artifact
  3) terminal chat (interactive check):
       llama-cli -m out/Qwen3-...-Q4_K_M.gguf -ngl 33 -c 4096

(Output formatting above is reconstructed against the current version; the numbers are the Run 2 measurements.)

A real run on an Apple M3 Pro. The "obviously best" Q8_0 failed the speed target, Q5_K_M missed by 0.41 tok/s, and the answer wasn't a quant alone — it was a quant plus the minimal GPU offload (-ngl 33, not the full 36). None of that is predictable from a spec sheet (full logs).

Install

Not on PyPI yet — install from git, into a virtualenv built with Python 3.11+ (macOS's system python3 is 3.9):

git clone https://github.com/leeyunseokarchive/fituna
python3.13 -m venv .venv && source .venv/bin/activate
pip install -e fituna

You also need llama.cpp, which provides the quantize/bench/perplexity engines FiTuna orchestrates:

brew install llama.cpp        # macOS/Linux Homebrew — ships all needed binaries
# or build from source (any platform):
git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B llama.cpp/build && cmake --build llama.cpp/build --config Release

Quickstart

Three ways in: a person runs the wizard; a script or CI job calls fituna run --json; an AI agent talks to fituna-mcp.

fituna quickstart

Six steps — environment check, targets, license requirements, model, quality corpus, search — and it prints the assembled fituna run ... command before executing it, so the next run is a one-liner you already have. It needs a terminal; in CI or a pipe use fituna run directly, since every search parameter maps to a public run flag (proven by an argv-equality test). Model download (a curated shortlist) and HuggingFace search are wizard-only conveniences; run --model expects a .gguf on disk.

It never predicts throughput: memory fit is arithmetic (published file size vs detected VRAM/RAM, assumed margin stated), speed is measured, and curated or HuggingFace-search candidates show their license — local-scan and manual-path options cannot, since a .gguf carries no license metadata. Any docs/RESULTS.md figure it cites is a record of what was measured on named hardware, never a prediction.

The script path (what the wizard assembles for you)

Quality loss is perplexity increase over a plain-text corpus, so it is only meaningful on text resembling your workload. Any UTF-8 file works (--quality-corpus); both presets are one command away:

fituna fetch-corpus --lang en --out wikitext-2-raw-test.txt        # wikitext-2 test split
fituna fetch-corpus --lang ko --out kowiki-corpus.txt --rows 500   # Korean Wikipedia

Measure the language you'll run: the same quant can measure 2–3× different loss on the two corpora, and in Run 3 that was enough to change the verdict the tool returned (measurement and caveats). Both presets are CC BY-SA 3.0 and fetch-corpus prints the license notice and source URL when it finishes; --dataset/--config/--split override the preset (provenance and licensing).

fituna doctor                             # confirm the environment is ready
fituna detect-hw                          # see what FiTuna detects
fituna run --model your-model-F16.gguf \
  --target-tps 30 --max-quality-loss 5 \
  --ctx 4096 --wikitext wikitext-2-raw-test.txt --out ./out --resume

Pass an F16/BF16 .gguf directly (many models publish one), or an HF-format directory if convert_hf_to_gguf.py is available (source checkout + pip install torch transformers; package-manager builds don't ship it).

Disk usage: the search quantizes every candidate reaching the quality stage — ~12 GB for four candidates of a 4B model. Files are reused across runs; narrow --quant to bound this.

Why

Running a local LLM means picking a quantization level (Q2–Q8), a GPU offload layer count (-ngl) and a context length — a search space people navigate today by trial and error:

• Ollama / LM Studio apply fixed per-model presets; a request for finer quantization control was closed as not planned.
• NVIDIA Model Optimizer's AutoQuantize is CUDA-only.
• VRAM calculators & chatbot advice estimate from specs — and specs don't know your thermals, memory bandwidth, or llama.cpp build flags.

FiTuna replaces the guesswork with a measured search over the llama.cpp binaries you already have — verified on your hardware, reproducible from cache.

Features

• 🔍 Target-driven search — in: model + target tok/s + max quality loss %. Out: quant × -ngl × ctx config + a ready-to-run command.
• 📏 Measured, not assumed — candidates walked in measured perplexity order (in our runs Q6_K beat Q8_0 — data), with a binary search for the minimal GPU offload.
• ⚡ Aggressive early exits — quality-gate failures and hopeless quants are skipped without wasting benches; a bench that can't finish in time counts as "too slow", not a crash.
• 🗃️ Reproducible cache — sqlite3, keyed by model fingerprint × hardware × llama.cpp build version; --resume re-answers in <1s and survives interruptions.
• 🖥️ Hardware auto-detection — NVIDIA (nvidia-smi), AMD (rocm-smi), Apple Silicon unified memory (system_profiler), with manual override.
• 🪶 Zero runtime dependencies — pure Python 3.11+ stdlib.

Measured results

Apple M3 Pro, llama.cpp build 9960. Full logs, timings and run-to-run variance analysis (including a thermal-throttle outlier we caught and documented): docs/RESULTS.md · Scenarios: docs/USE_CASES.md · Reproduce on NVIDIA/Linux with the one-click Colab notebook (free T4 tier): notebooks/colab_nvidia_verification.ipynb

How it works

Stage 1 measures perplexity loss for every candidate, because Stage 2 walks them in measured quality order and you can't sort by a number you haven't measured. Stage 2 early-exits hard: a quant missing the target at full offload is dropped without further benches, and the first quant that passes wins. Results cache to sqlite3 keyed by model fingerprint, hardware profile and llama.cpp build version, so --resume never serves numbers from a different backend build. Diagrams, module map and full algorithm: docs/ARCHITECTURE.md · Contract: fituna/config.py

Use as a library

Zero runtime dependencies means the modules import directly:

from fituna.hardware import detect_hardware

hw = detect_hardware()
print(f"{hw.gpu_vendor.value}: {hw.gpu_name}, {hw.vram_mb} MB VRAM, {hw.ram_mb} MB RAM")
# apple: Apple M3 Pro, 18432 MB VRAM, 18432 MB RAM

(Real output from a python3.13 -c run on the same M3 Pro as above.) Driving the search programmatically means calling fituna.search.search(), which also needs a ModelInfo, resolved BinaryPaths, a work directory and a corpus path — exactly what fituna run/quickstart assemble for you (search.py, config.py).

MCP server — measured answers for AI agents

Ask a chatbot "which local model config fits my machine?" and it guesses from specs. Point it at FiTuna's MCP server and it measures:

claude mcp add fituna -- fituna-mcp      # or any MCP client, stdio transport

Stdlib-only like the rest of FiTuna — MCP stdio is newline-delimited JSON-RPC 2.0, no SDK required (fituna/mcp_server.py).

Scope

FiTuna recommends; it doesn't execute or serve. The output is the quantized .gguf plus llama-server / llama-cli commands you copy and run (and, with --export-ollama, an Ollama Modelfile beside it) — FiTuna launches none of them. That's a deliberate boundary: serving inference is llama.cpp's job, and duplicating it would add no differentiated value (rationale). The two extensions that stay inside it — --launch and an LM Studio preset export — are tracked in #19.

Known limitations

• Single GPU only — first GPU reported by nvidia-smi/rocm-smi; no --tensor-split (#11, help wanted: we have no multi-GPU machine to measure on).
• Windows AMD auto-detection — rocm-smi has no mainstream Windows distribution; use --gpu amd --vram-mb <N>.
• Quality = perplexity on a corpus you choose — a proxy, not a guarantee of domain quality. Gate on text resembling your workload (--quality-corpus; measured EN-vs-KO comparison).
• The quality verdict depends on --ppl-chunks — loss is an estimate over chunks × 512 tokens whose absolute value grows with the chunk count, so re-measure a candidate close to your budget before trusting the PASS (measured effect). quality.py still parses the PPL and discards llama-perplexity's error bar (#8) — which is how Run 5 came to publish a claim it later had to withdraw.
• Benchmarks are thermally sensitive — verdicts within a few tok/s of the target are marginal (variance analysis).
• Real-hardware E2E covers macOS and Linux only — Apple Silicon/Metal and NVIDIA T4/CUDA. Windows paths are unit-tested and CI-run, but not yet integration-run against real binaries (#12).

Contributing

Contributions welcome — the codebase is small, dependency-free and contract-first (start at fituna/config.py); 246 unit tests, per-module self-checks and a 3-OS × 2-Python CI matrix guard it. The roadmap lives in the v0.2.0 milestone, including #10 (parser test coverage, good first issue). See CONTRIBUTING.md · docs/DEVELOPMENT.md · CHANGELOG.md · SECURITY.md.

License

MIT © FiTuna contributors. Third-party notices (llama.cpp and subprocess-invoked tools): THIRD_PARTY_NOTICES.md · SBOM: docs/SBOM.md · Open-source usage: docs/OPEN_SOURCE_USAGE.md · AI-assisted development disclosure: docs/AI_MODEL_USAGE.md