LLM FitLab

An evidence-based compatibility radar for running open LLMs on the hardware you actually have.

FitLab answers the three questions every engineer asks before pulling a local model — and it answers them with regenerated data, not a hand-maintained list. Every claim on the page is produced by a script, validated against a JSON Schema, and merged through a reviewed pull request.

Source of truth: data/registry.json · Design rationale: ARCHITECTURE.md

Table of contents

• Why FitLab exists
• Installation
• Quick start
• CLI reference
• Configuration
• Running FitLab inside an organization
• Data contract
• How the numbers are produced
• Automation
• Development
• Contributing
• Security
• Support and versioning
• Related projects
• License

Why FitLab exists

Curated "best model for X GB" lists go stale within a quarter. FitLab is built so that the data outlives any individual model:

1. Nothing is hand-ranked. A weekly job pulls the leading models per Hugging Face pipeline tag through the public API, runs each one through the FITS engine, scores them with a versioned formula, and opens a review PR. Humans review; automation refreshes.
2. Claims are tested, not copied. One model per week receives a real benchmark on a free T4 16 GB — the same GPU class the Colab free tier allocates — so the numbers transfer directly.
3. The community is the sensor network. Anyone can run the one-click Colab notebook or the local benchmark script; the result is schema-validated and sanity-bounded by CI before it is merged, with attribution.
4. Writes are gated. Every JSON is validated against data/schema/, and all automation writes through pull requests — never directly to the default branch.
5. Compatibility is re-verified, not assumed. A green PLUGS cell means the integration worked this week, at version pins recorded alongside the result.

Installation

Requirements: Python 3.10 or newer. Ollama is required only for the bench command and the wizard's benchmarking step — fit checks need no GPU and no runtime.

FitLab is published to PyPI from tagged GitHub releases via Trusted Publishing:

pip install fitlab

Until the first release lands on PyPI, install from source. The Makefile bootstraps uv, creates ./.venv and installs everything:

git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install
make run

Or without make, using any tool that reads pyproject.toml:

pip install .                     # or: uv sync
fitlab --version

The model list refreshes itself at runtime: the CLI fetches the weekly registry.json, caches it for 24 hours, and falls back to a bundled seed when offline. Upgrading the package is not required to see new models.

Quick start

Run the wizard with no arguments — it detects your hardware, shows what fits, and offers to benchmark your selection:

fitlab

// llm fitlab — fit-check console · v0.1.0
your rig: NVIDIA GeForce RTX 3060 · 12.0 GB VRAM · ~360 GB/s
registry 2026-08-24 · 22 models · live (weekly)

  1 colab-free — Top 10 — Colab free / Kaggle (T4 16 GB)
  2 local-12gb — Top 10 — local inference, low-end GPU (12 GB)
  3 multimodal — Top multimodal (vision-language) for 12–16 GB
  4 all        — everything, deduplicated
? Category 1-4 (1)

Or check a specific GPU profile without running anything:

fitlab check --gpu rtx3060-12 --category local-12gb --limit 6

                  fit check · RTX 3060 12GB · Q4_K_M · 8K ctx
┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ # ┃ model             ┃ params ┃ est GB ┃ verdict ┃ ~tok/s ┃ ollama tag      ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1 │ Qwen/Qwen3-8B     │ 8.2B   │ 7.2    │ FITS    │ ~44.8  │ qwen3:8b        │
│ 2 │ ibm-granite/gran… │ 8.1B   │ 7.3    │ FITS    │ ~45.4  │ granite3.3:8b   │
│ 3 │ meta-llama/Llama… │ 8.0B   │ 7.0    │ FITS    │ ~45.9  │ llama3.1:8b     │
│ 4 │ mistralai/Minist… │ 8.0B   │ 7.1    │ FITS    │ ~45.9  │ ministral-8b    │
│ 5 │ Qwen/Qwen3-14B    │ 14.8B  │ 11.5   │ TIGHT   │ ~24.8  │ qwen3:14b       │
│ 6 │ deepseek-ai/Deep… │ 14.8B  │ 11.8   │ TIGHT   │ ~24.8  │ deepseek-r1:14b │
└───┴───────────────────┴────────┴────────┴─────────┴────────┴─────────────────┘

After a benchmark, the CLI prints a prefilled submission link so your result can join the public dataset. Submitting is entirely optional; nothing is transmitted automatically.

CLI reference

fitlab check

fitlab bench

Benchmarks are seeded and fixed-length (fitlab-v1-256 prompt, num_predict 256, seed 42) so runs are comparable across machines and across time.

Configuration

All configuration is by environment variable — no config file, no implicit state beyond the cache.

Network behaviour is deliberately narrow and predictable:

• check, detect and the wizard's fit stage perform one HTTPS GET for the registry.
• Nothing is uploaded. Benchmark results are written to local disk; sharing is a manual step.
• With no network, the CLI degrades in a defined order: fresh cache → stale cache → bundled seed, and the active source is printed on every run.

Running FitLab inside an organization

Internal registry mirror. Fit verdicts and rankings are just data. To decouple from the public repository — for change control, or to publish an approved-model list — host your own registry.json and point clients at it:

export FITLAB_REGISTRY_URL="https://models.internal.example.com/fitlab/registry.json"
fitlab check --gpu rtx4060ti-16

Pinned, reproducible data. Because every registry carries schema_version, ranking_version and generated_at, you can pin clients to a specific commit and get byte-identical results:

export FITLAB_REGISTRY_URL="https://raw.githubusercontent.com/ruslanmv/fitlab/<commit-sha>/data/registry.json"

Air-gapped use. Ship a registry file with your image and reference it directly; no egress is required for fit checks:

export FITLAB_REGISTRY_URL="file:///opt/fitlab/registry.json"

Custom hardware fleet. Add your GPUs to data/hardware.yaml — including capability flags such as bf16, flash_attn2 and marlin — and regenerate the registry to get a fit matrix for your own fleet rather than the public reference profiles.

Extending the compatibility matrix. Add a component to data/stack.yaml with a probe that exercises it end to end, and it will be verified on the same weekly cadence as the rest of the stack.

Data contract

Everything downstream depends on data/, and only on data/. The site is a renderer; the CLI is a client. Both read the same documents.

data/
  registry.json              generated — the published source of truth (do not hand-edit)
  models.seed.yaml           human-curated seeds and editorial notes
  hardware.yaml              GPU profiles: VRAM, bandwidth, architecture capability flags
  stack.yaml                 ecosystem components and version-discovery rules
  benchmarks/                append-only measured results, one JSON per run
  schema/
    model.schema.json        registry model entries
    benchmark.schema.json    benchmark result submissions

registry.json top-level keys: schema_version, generated_at, ranking_version, models, categories, benchmarks_latest, compat.

Stability guarantees:

• Every document carries schema_version, so historical data stays parseable indefinitely.
• data/benchmarks/ is append-only; results are never rewritten or deleted.
• Models withdrawn from Hugging Face are marked stale and hidden after 60 days — never removed.
• Fit values are labeled estimated until a real benchmark upgrades them to measured.

How the numbers are produced

FITS is arithmetic, not opinion:

est_vram = weights(params × bpw / 8 × 1.03)
         + kv_cache(2 × layers × kv_heads × head_dim × context × 2 bytes)
         + runtime_overhead

Verdicts are thresholds against the profile's VRAM: fits at ≤ 92%, tight at ≤ 100%, offload at ≤ 160% (partial GPU offload viable), otherwise no. Mixture-of-experts models are sized by total parameters for memory and by active parameters for speed; the registry stores both.

RUNS measures load time, time-to-first-token, generation tokens/second and peak VRAM through Ollama's /api/generate timing fields — chosen because Ollama is what the target audience actually runs. The weekly rotation is week_number % queue_length, so every model is measured and periodically re-measured.

Ranking is a transparent, versioned scored sort (currently ranking_version 1.1):

score = 0.35·fit + 0.25·hf_momentum + 0.25·speed + 0.15·plugs

where speed uses a measured value when one exists and a bandwidth-model estimate discounted by 0.5× otherwise. Changing the weights requires a version bump, making every rank shift visible in the pull request diff.

Automation

Operational properties: workflows use concurrency groups to prevent overlapping writes, every script is idempotent and safe to re-run, and degradation is graceful — if Hugging Face is unreachable the sync PR is simply not opened and the last-good registry stands; if Kaggle is unavailable the CPU lane still lands a datapoint.

Development

The repository is driven by a Makefile backed by uv. From a clean clone, two commands give you a working environment and a running CLI — uv is bootstrapped automatically if it is not already installed:

git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install          # creates ./.venv and installs fitlab + all tooling
make run              # launches the fit-check wizard

make run depends on make install, so make run alone is enough on a fresh clone. Run make help to list every target.

Targets accept overrides on the command line:

make check GPU=t4-16 CATEGORY=colab-free LIMIT=5
make bench MODEL=qwen3:14b REPS=5

Available variables: GPU, QUANT, CTX, CATEGORY, LIMIT, MODEL, REPS, OUT.

In a checkout, the Makefile points FITLAB_REGISTRY_URL at the repository's own data/registry.json, so every target reads the registry you are editing and works offline. Override it to exercise a remote fetch:

make run FITLAB_REGISTRY_URL=https://example.com/fitlab/registry.json

Dependency groups live in pyproject.toml: the maintainer extra carries what the scripts/ engines need (pyyaml, jsonschema), and dev adds ruff, build and twine on top. uv.lock is committed, so make install resolves to identical versions on every machine.

The engines can also be invoked directly, without make:

# FITS — verdict for one model on one GPU
python scripts/estimate_vram.py --model Qwen/Qwen3-14B --gpu rtx3060-12 --quant Q4_K_M

# Registry refresh from Hugging Face (writes data/registry.json locally)
python scripts/sync_hf.py

# RUNS — benchmark on whatever hardware you are on (local, Colab, or Kaggle)
python scripts/bench_gguf.py --model qwen3:8b --out data/benchmarks/

# PLUGS — end-to-end stack probes (CPU is sufficient)
python scripts/compat_probe.py --report data/compat_report.json

# Schema validation
python scripts/validate_data.py

compat_probe.py and validate_data.py both exit with the number of failures, so CI fails honestly rather than reporting a green matrix built on skipped probes.

There is no unit-test suite yet; make test runs linting, schema validation and a CLI smoke test.

The published site is static and reads data/*.json directly; serve site/ through GitHub Pages or any static host.

Contributing

Benchmark results are the most valuable contribution and take about three minutes on free infrastructure:

1. Open colab/benchmark_colab.ipynb in Colab, select a T4 runtime, and run all cells — or run python scripts/bench_gguf.py --model <tag> --source community-local on your own machine.
2. The final cell prints your result and a link that opens a prefilled GitHub issue.
3. CI validates the JSON and merges it; your GPU and handle appear on the page.

Model suggestions belong in data/models.seed.yaml only — never in the generated registry.json. Ranks are computed rather than negotiated: to change a rank, change the formula in scripts/sync_hf.py and bump ranking_version. Full guidelines are in CONTRIBUTING.md.

Security

• Community submissions are data, never code. Submitted JSON is parsed and validated; it is never executed.
• Untrusted code paths are isolated. The weekly stack probes install unpinned latest releases and run inside ephemeral CI runners, optionally under MatrixLab Docker isolation for local maintainer runs (see docker-compose.matrixlab.yml).
• Secrets never reach fork-triggered workflows. pull_request_target is deliberately avoided.
• Publishing uses OIDC. PyPI Trusted Publishing means no long-lived API token exists to leak.

Please report suspected vulnerabilities privately through GitHub Security Advisories rather than in a public issue.

Support and versioning

This is a community open-source project maintained on a best-effort basis; no service level is implied. Questions and bug reports belong in GitHub Issues.

Three version numbers move independently, and only the first is a package concern:

• Package version (fitlab) follows semantic versioning.
• schema_version changes only for breaking data-format changes; consumers should pin it.
• ranking_version changes whenever scoring weights change, so ranking shifts are auditable.

Benchmark results are point-in-time measurements on specific hardware and driver versions. They are a comparative signal, not a guarantee of what your machine will do.

Related projects

• OllaBridge — one OpenAI-compatible URL for all your local models
• HomePilot — local-first chat, image and video studio
• GitPilot — Git workflow automation
• MatrixLab — Docker isolation sandbox used for untrusted probes
• Best-of-the-Best — the auto-curation pattern this repository generalizes

License

Code is licensed under Apache-2.0. Data in data/ and all published registries are licensed under CC-BY-4.0.

Benchmark results are point-in-time measurements. Models referenced remain the property of their respective authors and are subject to their own licenses.