LLM FitLab

An evidence-based compatibility radar for running open LLMs on the hardware you actually have.

FitLab answers the three questions every engineer asks before pulling a local model — and it answers them with regenerated data, not a hand-maintained list. Every claim on the page is produced by a script, validated against a JSON Schema, and merged through a reviewed pull request.

Source of truth: data/registry.json · Design rationale: ARCHITECTURE.md

Table of contents

• Why FitLab exists
• Installation
• Quick start
• CLI reference
• Configuration
• Running FitLab inside an organization
• Data contract
• How the numbers are produced
• Media lane
• Using FitLab from HomePilot
• Secrets
• Automation
• Development
• Contributing
• Security
• Support and versioning
• Related projects
• License

Why FitLab exists

Curated "best model for X GB" lists go stale within a quarter. FitLab is built so that the data outlives any individual model:

1. Nothing is hand-ranked. A weekly job pulls the leading models per Hugging Face pipeline tag through the public API, runs each one through the FITS engine, scores them with a versioned formula, and opens a review PR. Humans review; automation refreshes.
2. Claims are tested, not copied. One model per week receives a real benchmark on a free T4 16 GB — the same GPU class the Colab free tier allocates — so the numbers transfer directly.
3. The community is the sensor network. Anyone can run the one-click Colab notebook or the local benchmark script; the result is schema-validated and sanity-bounded by CI before it is merged, with attribution.
4. Writes are gated. Every JSON is validated against data/schema/, and all automation writes through pull requests — never directly to the default branch.
5. Compatibility is re-verified, not assumed. A green PLUGS cell means the integration worked this week, at version pins recorded alongside the result.

Installation

Requirements: Python 3.10 or newer. Ollama is required only for the bench command and the wizard's benchmarking step — fit checks need no GPU and no runtime.

FitLab is published to PyPI from tagged GitHub releases via Trusted Publishing:

pip install fitlab

Until the first release lands on PyPI, install from source. The Makefile bootstraps uv, creates ./.venv and installs everything:

git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install
make run

Or without make, using any tool that reads pyproject.toml:

pip install .                     # or: uv sync
fitlab --version

The model list refreshes itself at runtime: the CLI fetches the weekly registry.json, caches it for 24 hours, and falls back to a bundled seed when offline. Upgrading the package is not required to see new models.

Quick start

Run the wizard with no arguments — it detects your hardware, shows what fits, and offers to benchmark your selection:

fitlab

// llm fitlab — fit-check console · v0.1.0
your rig: NVIDIA GeForce RTX 3060 · 12.0 GB VRAM · ~360 GB/s
registry 2026-08-24 · 22 models · live (weekly)

  1 homepilot-12gb — HomePilot — most liked chat models on 12 GB
  2 homepilot-16gb — HomePilot — most liked chat models on 16 GB
  3 homepilot-vision — HomePilot — most liked vision models on 16 GB
  4 local-12gb — Top 10 — local inference, low-end GPU (12 GB)
  5 colab-free — Top 10 — Colab free / Kaggle (T4 16 GB)
  6 multimodal — Top multimodal (vision-language) for 12–16 GB
  7 all        — everything, deduplicated
? Category 1-4 (1)

Or check a specific GPU profile without running anything:

fitlab check --gpu rtx3060-12 --category local-12gb --limit 6

                  fit check · RTX 3060 12GB · Q4_K_M · 8K ctx
┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ # ┃ model             ┃ params ┃ est GB ┃ verdict ┃ ~tok/s ┃ ollama tag      ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1 │ Qwen/Qwen3-8B     │ 8.2B   │ 7.2    │ FITS    │ ~44.8  │ qwen3:8b        │
│ 2 │ ibm-granite/gran… │ 8.1B   │ 7.3    │ FITS    │ ~45.4  │ granite3.3:8b   │
│ 3 │ meta-llama/Llama… │ 8.0B   │ 7.0    │ FITS    │ ~45.9  │ llama3.1:8b     │
│ 4 │ mistralai/Minist… │ 8.0B   │ 7.1    │ FITS    │ ~45.9  │ ministral-8b    │
│ 5 │ Qwen/Qwen3-14B    │ 14.8B  │ 11.5   │ TIGHT   │ ~24.8  │ qwen3:14b       │
│ 6 │ deepseek-ai/Deep… │ 14.8B  │ 11.8   │ TIGHT   │ ~24.8  │ deepseek-r1:14b │
└───┴───────────────────┴────────┴────────┴─────────┴────────┴─────────────────┘

After a benchmark, the CLI prints a prefilled submission link so your result can join the public dataset. Submitting is entirely optional; nothing is transmitted automatically.

CLI reference

fitlab check

fitlab bench

Benchmarks are seeded and fixed-length (fitlab-v1-256 prompt, num_predict 256, seed 42) so runs are comparable across machines and across time.

Configuration

All configuration is by environment variable — no config file, no implicit state beyond the cache.

Network behaviour is deliberately narrow and predictable:

• check, detect and the wizard's fit stage perform one HTTPS GET per published ref until one answers (normally the first).
• Nothing is uploaded. Benchmark results are written to local disk; sharing is a manual step.
• With no network, the CLI degrades in a defined order: fresh cache → stale cache → bundled seed, and the active source is printed on every run. Until 2026-09 the default URL was an unreplaced placeholder (OWNER/llm-fitlab@main) that 404s, so every installed copy quietly served the bundled seed and reported bundled-seed; if you see that on a networked machine, upgrade.

Running FitLab inside an organization

Internal registry mirror. Fit verdicts and rankings are just data. To decouple from the public repository — for change control, or to publish an approved-model list — host your own registry.json and point clients at it:

export FITLAB_REGISTRY_URL="https://models.internal.example.com/fitlab/registry.json"
fitlab check --gpu rtx4060ti-16

Pinned, reproducible data. Because every registry carries schema_version, ranking_version and generated_at, you can pin clients to a specific commit and get byte-identical results:

export FITLAB_REGISTRY_URL="https://raw.githubusercontent.com/ruslanmv/fitlab/<commit-sha>/data/registry.json"

Air-gapped use. Ship a registry file with your image and reference it directly; no egress is required for fit checks:

export FITLAB_REGISTRY_URL="file:///opt/fitlab/registry.json"

Custom hardware fleet. Add your GPUs to data/hardware.yaml — including capability flags such as bf16, flash_attn2 and marlin — and regenerate the registry to get a fit matrix for your own fleet rather than the public reference profiles.

Extending the compatibility matrix. Add a component to data/stack.yaml with a probe that exercises it end to end, and it will be verified on the same weekly cadence as the rest of the stack.

Data contract

Everything downstream depends on data/, and only on data/. The site is a renderer; the CLI is a client. Both read the same documents.

data/
  registry.json              generated — the published source of truth (do not hand-edit)
  ollama_catalog.json        generated — the Ollama library index, every model and tag
  models.seed.yaml           human-curated seeds and editorial notes
  media.seed.yaml            image + video models: declared VRAM (with source), workflow cost, install ids
  media_registry.json        generated — the media lane's published document (do not hand-edit)
  hardware.yaml              GPU profiles: VRAM, bandwidth, architecture capability flags
  stack.yaml                 ecosystem components and version-discovery rules
  benchmarks/                append-only measured results, one JSON per run
  schema/
    model.schema.json        registry model entries
    benchmark.schema.json    benchmark result submissions
    media.schema.json        the media registry document

registry.json top-level keys: schema_version, generated_at, ranking_version, models, categories, benchmarks_latest, compat, calibration.

calibration records how the bandwidth estimate compares with the measurements collected so far, rather than tuning the constant to fit them. As of 2026-09-12 it reads 2.45× over two pairs (qwen3:0.6b on an Actions CPU runner, qwen3:8b on a Kaggle P100) — the estimate is a consistently optimistic ceiling, and the leaderboard says so. Two points is not a refit; it is a warning label that gets sharper each week.

Where the published registry lives

Publishing to a branch is deliberate: a review gate nobody can open is a freeze. Every weekly-sync run between 2026-08-31 and 2026-09-09 rebuilt the registry correctly and then failed at PR creation with "GitHub Actions is not permitted to create or approve pull requests", so the published data sat at its seed date while the job reported failure. Data refresh is not code — the branch publishes itself, and the PR still carries it into master for review. To enable the PR too, turn on Allow GitHub Actions to create and approve pull requests in Settings → Actions → General.

Stability guarantees:

• Every document carries schema_version, so historical data stays parseable indefinitely.
• data/benchmarks/ is append-only; results are never rewritten or deleted.
• Models withdrawn from Hugging Face are marked stale and hidden after 60 days — never removed.
• Fit values are labeled estimated until a real benchmark upgrades them to measured.
• ollama_tag is only ever a tag that exists in ollama_catalog.json. make validate fails the build on a tag that has left the Ollama library, so the page never tells you to run something that is not there.
• Every benchmark result must attach to a registry model by model_id or ollama_tag; an unattributable result fails make validate rather than being silently dropped on join.
• One Ollama tag belongs to one model. A speech or quantised variant that resolves onto a text model's tag has the tag cleared, with a warning in the sync log.
• license and capabilities on an auto-discovered entry are derived: license:* and capability tags come from the Hugging Face model card, the rest from repo naming (-Coder-, -VL-, R1, -A3B). Curated seed values always win.

How the numbers are produced

FITS is arithmetic, not opinion:

est_vram = weights(params × bpw / 8 × 1.03)
         + kv_cache(2 × layers × kv_heads × head_dim × context × 2 bytes)
         + runtime_overhead

Verdicts are thresholds against the profile's VRAM: fits at ≤ 92%, tight at ≤ 100%, offload at ≤ 160% (partial GPU offload viable), otherwise no. Mixture-of-experts models are sized by total parameters for memory and by active parameters for speed; the registry stores both.

RUNS measures load time, time-to-first-token, generation tokens/second and peak VRAM through Ollama's /api/generate timing fields — chosen because Ollama is what the target audience actually runs. The weekly rotation is week_number % queue_length, so every model is measured and periodically re-measured.

Ranking is a transparent, versioned sort (currently ranking_version 1.2, as computed in scripts/sync_hf.py). The published order is Hugging Face likes among models that fit the reference GPU. The score breaks ties:

score = 0.30·fit + 0.20·capability + 0.35·popularity + 0.10·speed + 0.05·plugs
popularity = 0.75·log likes + 0.25·log downloads

Likes are people's votes. Downloads only corroborate a model that has not been liked yet. capability is log-scaled parameter count, and speed uses a measured value when one exists and a bandwidth-model estimate discounted by 0.5× otherwise. HomePilot lists keep only models it can install. Changing the weights or the order requires a version bump, making every rank shift visible in the pull request diff.

MEDIA (image and video generation) has its own lane — see Media lane.

Secrets

All secrets live in GitHub — Settings → Secrets and variables → Actions → New repository secret. Nothing is configured on the Hugging Face side: a static Space has no secrets and needs none, because the page is a static file that fetches a public registry.json. The Space never calls Kaggle or Hugging Face with a token; the GitHub workflow holds the credentials, builds the data, and pushes the finished page to the Space.

KAGGLE_API_TOKEN is the variable the current Kaggle CLI reads and the only one that accepts a KGAT_… token; KAGGLE_KEY is interpreted as a legacy API key and fails against one. The workflow passes both, so an older KAGGLE_KEY still works if that is what you have.

Also enable Allow GitHub Actions to create and approve pull requests (Settings → Actions → General → Workflow permissions). Without it the weekly review PRs cannot be opened; data still publishes to registry-latest, so the leaderboard stays current either way.

GitHub repo secrets ──> weekly-sync ────────> registry-latest branch ──┐
                   │                                                   ├──> leaderboard + CLI
                   ├──> weekly-benchmark ──> Kaggle T4 ──> data/ ──────┘
                   └──> sync-hf-space ─────> Hugging Face Space (static, no secrets)

Automation

Operational properties: workflows use concurrency groups to prevent overlapping writes, every script is idempotent and safe to re-run, and degradation is graceful — if Hugging Face is unreachable the sync PR is simply not opened and the last-good registry stands; if Kaggle is unavailable the CPU lane still lands a datapoint.

Development

The repository is driven by a Makefile backed by uv. From a clean clone, two commands give you a working environment and a running CLI — uv is bootstrapped automatically if it is not already installed:

git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install          # creates ./.venv and installs fitlab + all tooling
make run              # launches the fit-check wizard

make run depends on make install, so make run alone is enough on a fresh clone. Run make help to list every target.

Targets accept overrides on the command line:

make check GPU=t4-16 CATEGORY=colab-free LIMIT=5
make bench MODEL=qwen3:14b REPS=5

Available variables: GPU, QUANT, CTX, CATEGORY, LIMIT, MODEL, REPS, OUT.

In a checkout, the Makefile points FITLAB_REGISTRY_URL at the repository's own data/registry.json, so every target reads the registry you are editing and works offline. Override it to exercise a remote fetch:

make run FITLAB_REGISTRY_URL=https://example.com/fitlab/registry.json

Dependency groups live in pyproject.toml: the maintainer extra carries what the scripts/ engines need (pyyaml, jsonschema), and dev adds ruff, build and twine on top. uv.lock is committed, so make install resolves to identical versions on every machine.

The engines can also be invoked directly, without make:

# FITS — verdict for one model on one GPU
python scripts/estimate_vram.py --model Qwen/Qwen3-14B --gpu rtx3060-12 --quant Q4_K_M

# Registry refresh from Hugging Face (writes data/registry.json locally)
python scripts/sync_hf.py

# RUNS — benchmark on whatever hardware you are on (local, Colab, or Kaggle)
python scripts/bench_gguf.py --model qwen3:8b --out data/benchmarks/

# PLUGS — end-to-end stack probes (CPU is sufficient)
python scripts/compat_probe.py --report data/compat_report.json

# Schema validation
python scripts/validate_data.py

compat_probe.py and validate_data.py both exit with the number of failures, so CI fails honestly rather than reporting a green matrix built on skipped probes.

There is no unit-test suite yet; make test runs linting, schema validation and a CLI smoke test.

The published site is static and reads data/*.json directly; serve site/ through GitHub Pages or any static host.

Contributing

Benchmark results are the most valuable contribution and take about three minutes on free infrastructure:

1. Open colab/benchmark_colab.ipynb in Colab, select a T4 runtime, and run all cells — or run python scripts/bench_gguf.py --model <tag> --source community-local on your own machine.
2. The final cell prints your result and a link that opens a prefilled GitHub issue.
3. CI validates the JSON and merges it; your GPU and handle appear on the page.

Model suggestions belong in data/models.seed.yaml only — never in the generated registry.json. Ranks are computed rather than negotiated: to change a rank, change the formula in scripts/sync_hf.py and bump ranking_version. Full guidelines are in CONTRIBUTING.md.

Security

• Community submissions are data, never code. Submitted JSON is parsed and validated; it is never executed.
• Untrusted code paths are isolated. The weekly stack probes install unpinned latest releases and run inside ephemeral CI runners, optionally under MatrixLab Docker isolation for local maintainer runs (see docker-compose.matrixlab.yml).
• Secrets never reach fork-triggered workflows. pull_request_target is deliberately avoided.
• Publishing uses OIDC. PyPI Trusted Publishing means no long-lived API token exists to leak.

Please report suspected vulnerabilities privately through GitHub Security Advisories rather than in a public issue.

Support and versioning

This is a community open-source project maintained on a best-effort basis; no service level is implied. Questions and bug reports belong in GitHub Issues.

Three version numbers move independently, and only the first is a package concern:

• Package version (fitlab) follows semantic versioning.
• schema_version changes only for breaking data-format changes; consumers should pin it.
• ranking_version changes whenever scoring weights change, so ranking shifts are auditable.

Benchmark results are point-in-time measurements on specific hardware and driver versions. They are a comparative signal, not a guarantee of what your machine will do.

Related projects

• OllaBridge — one OpenAI-compatible URL for all your local models
• HomePilot — local-first chat, image and video studio
• GitPilot — Git workflow automation
• MatrixLab — Docker isolation sandbox used for untrusted probes
• Best-of-the-Best — the auto-curation pattern this repository generalizes

License

Code is licensed under Apache-2.0. Data in data/ and all published registries are licensed under CC-BY-4.0.

Benchmark results are point-in-time measurements. Models referenced remain the property of their respective authors and are subject to their own licenses.

Media lane

Image and video generation models have no single config.json to size from, so FITS arithmetic does not apply. The media lane keeps FitLab's rules — every number has a recorded source, and nothing is hand-ranked — with a different input:

• data/media.seed.yaml declares each base checkpoint's VRAM need (min = runs with offload / low-VRAM mode, recommended = native resolution without offload), where each number came from (homepilot-presets, editorial, later measured), and the work its default workflow does (steps × megapixels × frames).
• scripts/build_media.py adds live Hugging Face likes and downloads, derives a verdict per GPU and ranks under media_ranking_version 1.1. Tops are ordered by likes and only include checkpoints HomePilot can install:

score = 0.40·fit + 0.25·capability + 0.25·momentum + 0.10·speed
momentum: 0.75·log likes + 0.25·log downloads
fit:  fits ≥ recommended · tight ≥ min · offload ≥ 0.7·min · otherwise no

The output, data/media_registry.json, publishes Top-10 lists for 12, 16 and 24 GB GPUs (image and video), and — so a client can rank for any VRAM without re-implementing the formula — each model's GPU-independent components (capability, momentum, speed) alongside the scoring weights. A client computes only the fit verdict for its own GPU: score = Σ weight · component, with fit = fit_score[verdict].

make media                         # refresh HF stats and rebuild
python scripts/build_media.py --offline   # rebuild from the seed, keeping previous stats

The LLM registry.json is unchanged by this lane: its schema, keys and consumers stay as they were.

Using FitLab from HomePilot

HomePilot reads FitLab to suggest the top models for the GPU it is running on — chat and vision from registry.json, image and video from media_registry.json — under Models → Suggested for your GPU. Suggestions are the models HomePilot can install that fit the machine, ordered by Hugging Face likes, so the first row is the one people have ranked highest. The contract it relies on:

Inference lane — hosted models, ranked by people

scripts/sync_inference.py publishes data/inference_registry.json: for each HomePilot model type (chat, vision, image, video) the models Hugging Face Inference Providers serve live for the task HomePilot calls (adapters and GGUF repos excluded), in five orders:

Every row keeps its raw likes, downloads and trending_score beside the score, plus the live_providers serving it. Each type also reports diversity.top_publisher_share, to watch one publisher crowding the list. Popularity guides; it does not measure quality on your task. The ranking is the pure module fitlab.inference; HomePilot ranks live Hub data with the same rules and falls back to this feed, then to the copy it ships with.

| media_registry.json scoring + models.*.components | Ranking for any VRAM, identical to the published lists | | generated_at, ranking_version, media_ranking_version | Shown with every suggestion |

HomePilot fetches when the user presses Fetch definitions — and, if the user leaves it on, in a daily background check — reading registry-latest first and master second with If-None-Match, so an unchanged registry costs a 304 and no download. Offline it falls back to its last cached copy, then to a snapshot bundled in the release — the same order as the fitlab CLI. FITLAB_REGISTRY_URL points it at a mirror.

NVIDIA GPU catalog

Search 85 hardware profiles across desktop and laptop GeForce generations, or enter a custom GPU and VRAM capacity in either web interface. The CLI shares the same offline inventory (fitlab gpus, fitlab check --gpu 'RTX 4060 Ti 8GB'). Monthly NVIDIA source refreshes open review PRs; unknown specs and source conflicts stay explicit. See GPU catalog usage and maintenance for sources, custom hardware, launch guidance, validation, deployment and rollback.