jetfit 0.2.3

LLM model advisor for NVIDIA Jetson and DGX Spark unified-memory devices

jetfit

LLM model advisor for NVIDIA Jetson and DGX Spark unified-memory devices.

Detects your Jetson hardware, scores LLM models across quality, speed, and memory fit, and tells you exactly which quantization level will run well on your device. Ships with an interactive TUI (default) and a CLI mode.

---

Install

pip install jetfit

or with uv:

uv tool install jetfit   # install globally
uvx jetfit               # run without installing

---

Usage

TUI (default)

jetfit

Launches the interactive terminal UI. The top bar shows your detected platform, available RAM, accelerator type, and JetPack version. Models are listed in a scrollable table sorted by composite score, with estimated tok/s, best quantization, memory %, and fit grade per row.

Normal mode

Key	Action
j / k	Navigate models
g	Jump to top / bottom (toggle)
Enter	Open detail view
p	Open plan mode
m	Mark / unmark model for compare
c	Open compare view (marked vs selected)
x	Clear all marks
v	Enter visual select mode
/	Focus search bar
r	Cycle provider (family) filter
b	Cycle size filter
f	Cycle fit filter
s	Cycle sort column
-	Flip sort direction
F	Open advanced filter popup
S	Open hardware simulation
A	Open advanced config (tune efficiency)
C	Open calibrate screen
d	Open download screen
t	Cycle theme
h	Open help
q	Quit

Visual mode (v)

Select a contiguous range of models for bulk comparison.

Key	Action
j / k	Extend selection
m	Mark selected model
c	Open compare view for selection
v / Esc	Exit visual mode

Detail view (Enter)

Shows full quant ladder for the selected model — size, KV cache, total memory, memory %, estimated tok/s, and fit grade for every quantization level. Navigate rows with j/k; the left panel updates to show specs for the highlighted quant.

Key	Action
j / k	Navigate quant rows
d	Open download screen for this model + quant
Esc / q	Back

Plan mode (p)

Estimates hardware requirements for a model config. Edit Context, Quant, and Target TPS fields.

Key	Action
Tab / j / k	Move between fields
Type	Edit current field
Esc / q	Exit plan mode

Calibrate view (C)

Measures real tok/s via llama-bench and saves a hardware-specific efficiency factor. Once calibrated, all speed estimates update and the system bar shows a ✓ cal badge.

Field	Description
Model path	Path to a .gguf file. Press e to open the file browser.
Params (B)	Parameter count — auto-detected from filename.
Quant	Quantization level — auto-detected from filename.
llama-bench	Path to llama-bench binary — auto-searched from PATH.
N generate	Tokens per run. 128 recommended; 256 for larger models.

Key	Action
Tab	Move between fields
e	Open file browser to select model
Enter	Run calibration
Esc / q	Back

Results are saved to ~/.config/jetfit/calibration.json.

File browser (e)

Keyboard-driven file/directory picker that opens as a popup overlay.

Key	Action
j / k	Move up / down
Enter / → / l	Enter directory or select file
← / h	Go to parent directory
Esc	Cancel

Download view (d)

Browse and download GGUF model files directly from HuggingFace. Shows ✔ public or ✖ token required badges per model. Files downloaded in the current session are marked with ✔ ~~strikethrough~~.

Key	Action
j / k	Navigate model list
Enter	Fetch available GGUF files for selected model
Ctrl+j / Ctrl+k	Move file cursor in the file list
d	Download file at cursor (confirms overwrite if file exists)
e	Browse save directory
Tab	Move between save dir / token fields
Esc / q	Back

HF Token: Required for gated models (Llama, Gemma, etc.). Enter once and it is saved to ~/.config/jetfit/hf_token. The field is masked when not focused. Obtain a token at huggingface.co/settings/tokens.

Save directory: Defaults to ~/. Changed via the field or e to browse.

Progress is shown as a live bar during download: [████████░░░░░░░░░░░░] 42% 430 / 1024 MB. If the target file already exists, a warning is shown and a second d confirms overwrite.

Compare view (c)

Side-by-side comparison of marked models. Rows are attributes (Score, tok/s, Fit, Mem%, Params, Quant, Context); columns are models. Best values are highlighted.

Hardware simulation (S)

Override the active hardware profile to preview recommendations for any supported device. The system bar shows (sim) when active.

Advanced config (A)

Tune the efficiency factor used for tok/s estimation. Changes apply immediately.

Advanced filter (F)

Set numeric bounds on parameter count and memory utilization %.

---

CLI

# Detect hardware
jetfit system

# Detect hardware (JSON)
jetfit system --json

# Recommend models for current hardware
jetfit recommend

# Filter by model name
jetfit recommend --model llama

# Fix a specific quant level
jetfit recommend --quant Q4_K_M

# Show all quant levels per model
jetfit recommend --all-quants

# Override available memory
jetfit recommend --available-gb 12.0

# Target a specific hardware profile
jetfit recommend --profile jetson_agx_orin_64gb

# Minimum tok/s threshold
jetfit recommend --min-tps 5.0

# JSON output
jetfit recommend --json

# Calibrate tok/s estimates against real hardware (requires llama.cpp)
jetfit calibrate ~/models/Llama-3.1-8B-Q4_K_M.gguf

# With explicit options
jetfit calibrate model.gguf --params-b 8.0 --quant Q4_K_M
jetfit calibrate model.gguf --llama-bench /path/to/llama-bench

llama-bench is auto-searched from PATH. To build llama.cpp with CUDA support:

git clone https://github.com/ggml-org/llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j$(nproc)

---

Supported Hardware

Device	RAM	Bandwidth	Accelerator	JetPack
Jetson Nano	4 GB	25.6 GB/s	DLA+CUDA	4.x
Jetson TX2 NX	4 GB	51.2 GB/s	CUDA	5.x
Jetson TX2 4GB	4 GB	51.2 GB/s	CUDA	4.x
Jetson TX2	8 GB	59.7 GB/s	CUDA	4.x
Jetson TX2i	8 GB	51.2 GB/s	CUDA	4.x
Jetson Xavier NX 8GB	8 GB	59.7 GB/s	DLA+CUDA	5.x
Jetson Xavier NX 16GB	16 GB	59.7 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 16GB	16 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 32GB	32 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 64GB	64 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier Industrial	64 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson Orin Nano 4GB	4 GB	51.2 GB/s	CUDA	6.x
Jetson Orin Nano 8GB	8 GB	102.4 GB/s	CUDA	6.x
Jetson Orin NX 8GB	8 GB	102.4 GB/s	DLA+CUDA	6.x
Jetson Orin NX 16GB	16 GB	102.4 GB/s	DLA+CUDA	6.x
Jetson AGX Orin 32GB	32 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Orin 64GB	64 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Orin Industrial	64 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Thor T4000	64 GB	273 GB/s	FP4+CUDA	7.x
Jetson AGX Thor T5000	128 GB	273 GB/s	FP4+CUDA	7.x
DGX Spark (GB10)	128 GB	273 GB/s	FP4+CUDA	—

On macOS or Linux dev machines, jetfit runs in simulation mode — pick any profile with S.

---

How it works

1. Hardware detection — Reads device-tree model and compatible strings (/proc/device-tree/), tegra release (/etc/nv_tegra_release), and available RAM via tegrastats, jtop, or /proc/meminfo. On non-Jetson machines, falls back to simulation mode.

2. Model database — ~70 models embedded in fit.py. Each entry has parameter count, context length, and HuggingFace repo ID. Memory requirements are computed across a 6-level quantization ladder (Q8_0 through Q2_K).

3. KV cache accounting — Memory estimates include a fp16 KV cache and 0.5 GB runtime overhead, so "fits" means the model will actually load at a typical 4K inference context.

4. FP4 halving — On devices with FP4 support (Thor, DGX Spark), effective model size is halved before all memory and speed calculations.

5. Fit levels — Based on (weights + KV cache + overhead) / available_memory:

   Level	Utilization
   Perfect	≤ 70%
   Good	71–90%
   Marginal	91–100%
   TooTight	> 100%

6. Speed estimation — Token generation is memory-bandwidth-bound:

   (bandwidth_GB_s / effective_size_GB) × efficiency × quant_speed_multiplier

   Default efficiency is 0.50–0.55 per profile, tunable via A. Quant multipliers range from 1.00× (Q8_0) to 1.80× (Q2_K).

7. Composite score — 0–100 combining normalized speed (45%), fit level (35%), and quantization quality (20%).

8. Calibration — Measures real tok/s via llama-bench, back-calculates an efficiency factor, and saves to ~/.config/jetfit/calibration.json.

9. Download — Fetches GGUF file listings via huggingface_hub and streams downloads with live progress. HF token saved to ~/.config/jetfit/hf_token.

---

Versioning

MAJOR.MINOR.PATCH (SemVer):

Change type	Version bump
Breaking changes (CLI/API incompatibility)	MAJOR
New features	MINOR
Bug fixes, hotfixes	PATCH

---

Project structure

jetfit/
  __init__.py      -- version
  cli.py           -- Click CLI entry point, TUI launch
  hardware.py      -- Jetson/DGX hardware detection
  profiles.py      -- Hardware profile database (22 devices)
  fit.py           -- Scoring engine, quantization ladder, model catalog (~70 models)
  calibration.py   -- llama-bench runner, efficiency back-calculation, calibration I/O
  downloader.py    -- HuggingFace GGUF file listing and streaming download
  tui.py           -- Textual TUI (app state, rendering, keyboard events)
tests/
  test_hardware.py
  test_fit.py
  test_calibration.py
  test_ros2.py
pyproject.toml
LICENSE

---

Dependencies

Package	Purpose
click	CLI argument parsing
rich	CLI table and colored output
textual	Terminal UI framework
huggingface_hub	HuggingFace model metadata and file listing

---

License

MIT