jetfit 0.1.6

LLM model advisor for NVIDIA Jetson and DGX Spark unified-memory devices

jetfit

LLM model advisor for NVIDIA Jetson and DGX Spark unified-memory devices.

Detects your Jetson hardware, scores LLM models across quality, speed, and memory fit, and tells you exactly which quantization level will run well on your device.

Ships with an interactive TUI (default) and a CLI mode. Supports hardware simulation, calibration, compare view, and plan mode.

---

Install

pip install jetfit

or with uv:

uv tool install jetfit   # install globally
uvx jetfit               # run without installing

---

Usage

TUI (default)

jetfit

Launches the interactive terminal UI. The top bar shows your detected platform, available RAM, accelerator type, and JetPack version (detected from /etc/nv_tegra_release on real hardware; minimum supported version in simulation mode). Models are listed in a scrollable table sorted by composite score by default, with estimated tok/s, best quantization, memory %, and fit grade per row.

Normal mode

Key	Action
j / k	Navigate models
g	Jump to top / bottom (toggle)
Enter	Open detail view
p	Open plan mode
m	Mark / unmark model for compare
c	Open compare view (marked vs selected)
x	Clear all marks
v	Enter visual select mode
/	Focus search bar
r	Cycle provider (family) filter
b	Cycle size filter
f	Cycle fit filter
s	Cycle sort column
-	Flip sort direction
F	Open advanced filter popup
S	Open hardware simulation
A	Open advanced config (tune efficiency)
C	Open calibrate screen
t	Cycle theme
h	Open help
q	Quit

Visual mode (v)

Select a contiguous range of models for bulk comparison.

Key	Action
j / k	Extend selection
m	Mark selected model
c	Open compare view for selection
v / Esc	Exit visual mode

Detail view (Enter)

Shows full quant ladder for the selected model — size, KV cache, total memory, memory %, estimated tok/s, and fit grade for every quantization level. Navigate rows with j/k; the left panel updates to show specs for the highlighted quant.

Plan mode (p)

Estimates hardware requirements for a model config. Edit Context, Quant, and Target TPS fields. Shows minimum and recommended RAM, feasibility per run path, and upgrade deltas.

Key	Action
Tab / j / k	Move between fields
Type	Edit current field
Backspace	Remove characters
Esc / q	Exit plan mode

Calibrate view (C)

Measures real tok/s via llama-bench and saves a hardware-specific efficiency factor for the current profile. Once calibrated, all model speed estimates are updated automatically and the system bar shows a ✓ cal badge.

Field	Description
Model path	Path to a .gguf file. Press Ctrl+O to open the file browser.
Params (B)	Parameter count in billions — auto-detected from filename (e.g. 8B).
Quant	Quantization level — auto-detected from filename (e.g. Q4_K_M).
llama-bench	Path to the llama-bench binary — auto-searched from PATH.
N generate	Tokens to generate per run. 128 is recommended; use 256 for larger models.

Press Enter to run. Results are saved to ~/.config/jetfit/calibration.json.

Key	Action
Tab / j / k	Move between fields
Ctrl+O	Open file browser to select model
Enter	Run calibration
Esc / q	Back

File browser (Ctrl+O)

Keyboard-driven .gguf file picker that opens as a popup over the calibrate screen.

Key	Action
j / k	Move up / down
Enter / → / l	Enter directory or select file
← / h	Go to parent directory
Esc	Cancel

Compare view (c)

Side-by-side comparison of marked models. Rows are attributes (Score, tok/s, Fit, Mem%, Params, Quant, Context); columns are models. Best values are highlighted.

Hardware simulation (S)

Override the active hardware profile to preview recommendations for any supported Jetson or DGX Spark device without leaving the TUI. The system bar shows (sim) when active.

Advanced config (A)

Tune the efficiency factor used for tok/s estimation. Changes apply immediately and all scores are recalculated.

Advanced filter (F)

Set numeric bounds on parameter count and memory utilization %.

---

CLI

# Detect hardware
jetfit system

# Detect hardware (JSON)
jetfit system --json

# Recommend models for current hardware
jetfit recommend

# Filter by model name
jetfit recommend --model llama

# Fix a specific quant level
jetfit recommend --quant Q4_K_M

# Show all quant levels per model
jetfit recommend --all-quants

# Override available memory
jetfit recommend --available-gb 12.0

# Target a specific hardware profile
jetfit recommend --profile jetson_agx_orin_64gb

# Minimum tok/s threshold
jetfit recommend --min-tps 5.0

# JSON output
jetfit recommend --json

# Calibrate tok/s estimates against real hardware (requires llama.cpp)
jetfit calibrate ~/models/Llama-3.1-8B-Q4_K_M.gguf

# With explicit options
jetfit calibrate model.gguf --params-b 8.0 --quant Q4_K_M
jetfit calibrate model.gguf --llama-bench /path/to/llama-bench

Calibration measures real tok/s via llama-bench, back-calculates an efficiency factor for the current hardware profile, and saves it to ~/.config/jetfit/calibration.json. All model speed estimates are updated on the next jetfit launch.

llama-bench is auto-searched from PATH. If not found, pass the path explicitly with --llama-bench. To build llama.cpp with CUDA support:

git clone https://github.com/ggml-org/llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j$(nproc)

The same calibration flow is available interactively via C in the TUI.

---

Supported Hardware

Device	RAM	Bandwidth	Accelerator	JetPack
Jetson Nano	4 GB	25.6 GB/s	DLA+CUDA	4.x
Jetson TX2 NX	4 GB	51.2 GB/s	CUDA	5.x
Jetson TX2 4GB	4 GB	51.2 GB/s	CUDA	4.x
Jetson TX2	8 GB	59.7 GB/s	CUDA	4.x
Jetson TX2i	8 GB	51.2 GB/s	CUDA	4.x
Jetson Xavier NX 8GB	8 GB	59.7 GB/s	DLA+CUDA	5.x
Jetson Xavier NX 16GB	16 GB	59.7 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 16GB	16 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 32GB	32 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier 64GB	64 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson AGX Xavier Industrial	64 GB	136.5 GB/s	DLA+CUDA	5.x
Jetson Orin Nano 4GB	4 GB	51.2 GB/s	CUDA	6.x
Jetson Orin Nano 8GB	8 GB	102.4 GB/s	CUDA	6.x
Jetson Orin NX 8GB	8 GB	102.4 GB/s	DLA+CUDA	6.x
Jetson Orin NX 16GB	16 GB	102.4 GB/s	DLA+CUDA	6.x
Jetson AGX Orin 32GB	32 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Orin 64GB	64 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Orin Industrial	64 GB	204.8 GB/s	DLA+CUDA	6.x
Jetson AGX Thor T4000	64 GB	273 GB/s	FP4+CUDA	7.x
Jetson AGX Thor T5000	128 GB	273 GB/s	FP4+CUDA	7.x
DGX Spark (GB10)	128 GB	273 GB/s	FP4+CUDA	—

On macOS or Linux dev machines, jetfit runs in simulation mode — pick any profile with S to preview recommendations.

---

How it works

1. Hardware detection — Reads device-tree model and compatible strings (/proc/device-tree/), tegra release (/etc/nv_tegra_release), and available RAM via tegrastats, jtop, or /proc/meminfo (priority order). On non-Jetson machines, falls back to simulation mode with a selectable profile.

2. Model database — 67 models embedded directly in fit.py. Each entry has a parameter count and real context length sourced from HuggingFace. Memory requirements are computed across a 6-level quantization ladder (Q8_0 through Q2_K) using per-quant bytes-per-parameter values that account for k-quant codebook overhead.

3. KV cache accounting — Memory estimates include a fp16 KV cache (0.000008 × params_b × 4096 GB) and 0.5 GB runtime overhead, so "fits" means the model will actually load at a typical 4K inference context.

4. FP4 halving — On devices with FP4 support (Thor, DGX Spark), effective model size is halved before all memory and speed calculations.

5. Fit levels — Based on (weights + KV cache + overhead) / available_memory:

   Level	Utilization
   Perfect	≤ 70%
   Good	71–90%
   Marginal	91–100%
   TooTight	> 100%

6. Speed estimation — Token generation is memory-bandwidth-bound. Estimated tok/s:

   (bandwidth_GB_s / effective_size_GB) × efficiency × quant_speed_multiplier

   Default efficiency is 0.50–0.55 per profile, tunable via A. Quant multipliers range from 1.00× (Q8_0) to 1.80× (Q2_K).

7. Composite score — Each model gets a 0–100 score combining normalized speed (45%), fit level (35%), and quantization quality (20%). Used for sorting and the score column.

8. Calibration — Run jetfit calibrate <model.gguf> to measure real tok/s via llama-bench and back-calculate an efficiency factor for the current hardware profile. Saved to ~/.config/jetfit/calibration.json. Once calibrated, all 67 model speed estimates are updated automatically and the system bar shows a ✓ cal badge.

---

Project structure

jetfit/
  __init__.py      -- version
  cli.py           -- Click CLI entry point, TUI launch
  hardware.py      -- Jetson/DGX hardware detection
  profiles.py      -- Hardware profile database (22 devices)
  fit.py           -- Scoring engine, quantization ladder, model catalog
  calibration.py   -- llama-bench runner, efficiency back-calculation, calibration I/O
  tui.py           -- Textual TUI (app state, rendering, keyboard events)
tests/
  test_hardware.py -- Hardware detection and TUI markup regression tests
  test_fit.py      -- Scoring engine unit tests
  test_calibration.py
  test_ros2.py
pyproject.toml
LICENSE

---

Dependencies

Package	Purpose
click	CLI argument parsing
rich	CLI table and colored output
textual	Terminal UI framework

---

License

MIT