localfit

One command for local, Kaggle, and cloud inference — and an honest answer to "will it fit?"

pip install localfit

Both localfit and llmrun are installed as commands and do the same thing.

Run any model, wherever it fits. Fits locally? Run it. Doesn't fit? Free Kaggle GPU. Still too big? RunPod. One command downloads the model, starts the server, configures your tools and launches the UI.

Will it fit?

The question every local-LLM guide gets wrong. On Apple Silicon your Mac's RAM is not what the GPU gets — Metal hands it roughly 74 %:

Check your own machine: ioreg -l | grep recommendedMaxWorkingSetSize, divide by 2³⁰.

A 19.5 GB model on a 24 GB Mac is 110 % of budget. It does not fail politely — it kernel-panics the machine. localfit scores every quant in a repo against the budget that actually exists:

localfit qwen3.8-27b                 # score against THIS machine
localfit qwen3.8-27b --gpu-gb 12     # score against a 12 GB card
localfit qwen3.8-27b --gpu-gb 17.8   # ...or someone else's 24 GB Mac

--gpu-gb is how you answer the question for hardware you do not own.

Vision projectors (mmproj-*, *-vision-*.gguf) are excluded from quant scoring — unfiltered, a 0.9 GB projector parses as an F16 quant, scores quality 100, and beats every real quant in the repo.

One Command — Everything Works

# Chat + image gen in Open WebUI (gemma4 LLM + Flux Klein 4B images)
llmrun launch openwebui --model gemma4:e4b --img klein-4b

# Code with image gen in localcoder
llmrun launch localcoder --model gemma4:e4b --img klein-4b

# Claude Code with image MCP tools
llmrun launch claude --model gemma4:e4b --img klein-4b

# Model doesn't fit locally? Run on free Kaggle GPU
llmrun launch openwebui --model gemma4:26b --remote kaggle

# Need more power? RunPod cloud ($0.22/hr)
llmrun launch openwebui --model gemma4:26b --img klein-9b --remote runpod --budget 30m

llmrun auto-detects your hardware, downloads models, installs Ollama if needed, starts the image server, configures Open WebUI with image generation enabled, and opens your browser. Zero manual config.

The Run Menu

llmrun run gemma4:e4b

Interactive menu with arrow key navigation — pick your backend before anything downloads:

╭──────────────────────── gemma-4-E4B ────────────────────────╮
│   LOCAL                                                     │
│   › MLX   gemma-4-E4B-4bit                      3.5GB      │
│     GGUF  Q4_K_M                                4.4GB      │
│     GGUF  Q8_0                                  7.5GB      │
│                                                             │
│   REMOTE                                                    │
│     Kaggle  T4 16GB                               free      │
│     RunPod  RTX 3090 24GB                     $0.22/hr      │
╰──────────────────────── Apple Silicon 24GB ─────────────────╯

Image Generation

One server, two APIs — works with Open WebUI, Claude Code, and any OpenAI-compatible client.

# Starts automatically with --img flag, or manually:
python -m llmrun.image_server 8189 klein-4b 4

The server implements OpenAI-compatible POST /v1/images/generations and multipart POST /v1/images/edits. llmrun configures both Open WebUI engines automatically when --img is present.

Supported Image Models

All models auto-detected via DiffusionPipeline.from_pretrained().

MCP Server for Claude Code

claude mcp add llmrun-image --transport stdio -- python3 -m llmrun.mcp_image

6 tools available:

Benchmarks (Verified)

LLM Backends

Ollama (Default)

llmrun run gemma4:e4b               # auto-installs Ollama, pulls model, serves

MLX (Apple Silicon)

pip install mlx-lm
llmrun run gemma-3-4b-it            # auto-picks mlx-community model

GGUF (llama.cpp)

llmrun run gemma4:26b               # downloads best GGUF quant for your GPU

Remote Serving

Kaggle (Free — 30h/week GPU)

llmrun login kaggle
llmrun run gemma4:e4b --remote kaggle

RunPod (Paid — any GPU)

llmrun login runpod
llmrun run gemma4:26b --cloud --budget $2

Combined Remote Chat + Diffusers

llmrun launch openwebui \
  --model HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive \
  --quant Q8_K_P \
  --img Qwen/Qwen-Image-Edit-2511 \
  --lora ScottzillaSystems/qwen-image-edit-plus-nsfw-lora \
  --remote runpod \
  --budget 30m

Text and image workloads use separate right-sized RunPod pods and separate state files. Each pod receives the same absolute provider-side deadline, an in-pod deadline guard, and the scheduled host reaper. Open WebUI is launched only after both OpenAI-compatible endpoints pass health checks. llmrun --stop discovers and terminates every managed pod, including sessions whose local state vanished.

This is full-weight Diffusers loading; no image quantization is silently applied. GPU cost is shown and approved separately for each pod.

ComfyUI on Kaggle or RunPod

# Current official ComfyUI + Manager on a free Kaggle T4, auto-stop after 15m
llmrun comfyui --remote kaggle --duration 15

# Large video/audio profile on RunPod; GPU and exact maximum cost are confirmed
llmrun comfyui MiniMaxAI/MiniMax-H3 --cloud --budget 30m

ComfyUI is a separate visual workflow runtime, not an OpenAI image endpoint. RunPod ComfyUI pods use the same provider terminateAfter, in-container guard, five-minute launchd reaper, and provider orphan discovery as text and Diffusers pods. Kaggle uses current official ComfyUI and its built-in Manager instead of the obsolete PyTorch/ComfyUI pins in older community notebooks.

MiniMax H3 is video plus native stereo audio, not a chat or still-image model. It cannot fit Kaggle's API-addressable 16GB GPU and is routed to an 80GB RunPod profile. Because H3 is a multi-file workflow, llmrun starts ComfyUI and leaves asset placement to its maintained H3 template/Manager instead of guessing paths.

Rebrand and compatibility

llmrun is the primary package, executable, UI name, and documentation name as of 2.0. The deprecated localfit executable and Python implementation package remain during the migration. Existing ~/.localfit state, RunPod environment metadata, ntfy topics, and tool-provider IDs are intentionally retained so old pods and configurations remain recoverable. Do not delete that directory while a legacy session may still exist.

Launch Any Tool

All tools are auto-configured with the right endpoints. Env vars scoped to subprocess — your normal setup is never touched.

Make It Fit — Remote Quantization

llmrun makeitfit Qwen2.5-7B-Instruct
# → Quantizes on Kaggle (free) or RunPod
# → Uploads to your HuggingFace repo
# → Run it: llmrun run yourname/model-Q4_K_M-GGUF-llmrun

All Commands

# Model management
llmrun run MODEL                    # interactive menu → serve
llmrun run MODEL --remote kaggle    # serve on free Kaggle GPU
llmrun run MODEL --cloud            # serve on RunPod
llmrun run MODEL --img IMAGE_MODEL  # serve LLM + image model
llmrun show MODEL                   # quants + fit analysis + pricing
llmrun list                         # installed models
llmrun stop                         # stop servers

# Image generation
llmrun launch openwebui --img klein-4b    # with Open WebUI
llmrun launch claude --img klein-4b       # with Claude Code MCP

# Tools
llmrun launch TOOL --model MODEL    # serve + launch tool
llmrun doctor                       # check all tool configs

# Hardware
llmrun                              # GPU dashboard + trending models
llmrun health                       # VRAM, temp, processes
llmrun bench                        # benchmark models
llmrun simulate                     # "will this model fit?"

# Cloud
llmrun login kaggle                 # save credentials
llmrun login runpod                 # save API key
llmrun login huggingface            # save HF token
llmrun comfyui --remote kaggle --duration 15
llmrun comfyui MODEL --cloud --budget 30m
llmrun --remote-status              # check Kaggle session
llmrun --remote-stop                # stop remote session

# System
llmrun check                        # check prerequisites
llmrun cleanup                      # free GPU memory
llmrun makeitfit MODEL              # quantize remotely

Supported Platforms

Requirements

• Python 3.10+
• Optional: mflux for Mac image gen
• Optional: Ollama (auto-installed by llmrun)

pip install llmrun                  # core
pip install mflux                     # + image generation (Mac)

License

Apache-2.0