MLX_Commander 🚀

The Norton Commander-style TUI, Headless CLI & Model Context Protocol (MCP) Orchestrator for Apple Silicon MLX Fine-Tuning Runs (Single Runs & Multi-Run Sweeps) and Hugging Face Dataset Preparation.

Built entirely in Python with zero mandatory dependencies and zero pre-compiled binaries.

Orthodox dual-panel TUI which makes Apple MLX hyperparameters and hardware defaults explicit

🌟 Key Capabilities at a Glance

• Fine-Tuning Single Run Orchestration (Mode 2):

  • Configure all 22 MLX fine-tuning parameters with explicit production defaults: iters, batch_size, gradient_accumulation_steps, learning_rate, lora_rank, lora_alpha, lora_dropout, max_seq_length, grad_checkpoint, mask_prompt, num_layers, save_every, and steps_per_eval.
  • Supports both Causal LLMs (mlx-lm) and Vision-Language / Multimodal models (mlx-vlm like Gemma 4, Qwen2-VL, PaliGemma, SmolVLM).
  • Automatic VLM Attention Mask Safeguards: Detects multimodal models and auto-adjusts micro-batch_size=1 and gradient_accumulation_steps=N to eliminate attention mask shape broadcast crashes.
  • Live Validation Loss Milestones: Discovers valid.jsonl and evaluates validation loss at regular intervals, highlighting milestones in the terminal (★ [Validation Loss Milestone] Iter 100: Val loss = 1.234).
  • Analytical Unified Memory (RAM) Estimator: Real-time calculation of peak RAM with [SAFE], [TIGHT], and [OOM RISK] safety ratings based on your exact Apple Silicon chip.
  • Implied Epochs Calculation: Automatically computes (iters * batch_size * grad_accumulation_steps) / train_records with dark red alerts when $< 1.0$.
  • Deterministic Checkpoint Naming: Adapters are automatically saved with complete hyperparameter signatures: 0000400_adapters_lora_r16_a32_lr1e-5_b4_i1000_Llama-3.2-3B-Instruct-4bit.safetensors.

• Fine-Tuning Multi-Run Sweeps & Queue Orchestration (Mode 3):

  • Single-Line Multi-Value Fields: Configure multiple sweep conditions on a single line (e.g. Learning Rate: [ 1e-4 ] [ 2e-4 ], LoRA Rank: [ 8 ] [ 16 ]).
  • Visual Cartesian Sweep Grid: Renders an interactive bottom panel displaying parameter columns, vertically stacked conditions, centered ✖ multiplication symbols, and total scheduled run count ($N_1 \times N_2 \dots$).
  • Sequential FIFO Queue Execution (mlx_commander --run-queue): Strictly enforces sequential execution to protect Unified Memory from thrashing, swap exhaustion, and macOS SIGKILL kernel panics.
  • Run Management: Add (F6), clone (c), delete (d), clear (x), or load (Enter) runs to tweak parameters on the fly.

• Generative Test Set Evaluation Engine (experimental):

  • Evaluates real token-by-token generation on test.jsonl deterministically (unlike standard mlx_lm.lora --test which only measures teacher-forced perplexity).
  • Physics-Grounded Throughput Engine: Computes model-dependent generation throughput ($\text{tok/s} \propto \frac{\text{Memory Bandwidth}}{\text{Model Active Weights}}$) scaling inversely with model parameter count ($1\text{B} > 3\text{B} > 8\text{B} > 70\text{B}$) and precision (4-bit vs fp16).
  • Live Milestone & Dynamic ETA: Real-time \r terminal progress updates (Sample 25/50 (50%) | 148.4 tok/s | ETA: 18s) with continuous cursor movement.
  • Deterministic Metrics: Exact Match (Strict & SQuAD-normalized), Substring Contains, and Word-Level F1 (Precision & Recall).
  • Matrix of Wrong Answers: Categorical Confusion Matrix & 2×2 Model Migration / Regression Matrix (FIXED, REGRESSED, PRESERVED, PERSISTENT_FAIL).
  • 3-Tier Storage Artifacts: Master eval_leaderboard.csv (opens in Apple Numbers/Excel), per-run eval_summary.json & eval_predictions.jsonl, and standalone zero-dependency offline eval_comparison.html dashboard.
  • Weights & Biases (W&B) Logging: Streams training loss and uploads interactive test prediction tables (wandb.Table) with prompt diffs and confusion heatmaps.

• Dataset Preparation & Standardization (Mode 1):

  • Converts Hugging Face datasets into 4 standardized Apple MLX formats:
    1. prompt_completion: {"prompt": "...", "completion": "..."} (mlx_lm.lora --mask-prompt compatible).
    2. chat: {"messages": [{"role": "system|user|assistant", "content": "..."}]} (multi-turn instruction tuning).
    3. text: {"text": "..."} (causal LM / continued pre-training).
    4. dpo: {"prompt": "...", "chosen": "...", "rejected": "..."} (Direct Preference Optimization).
  • Multi-column concatenation (+) with user ordering (e.g. instruction + input).
  • Multi-file selection and dataset merging with reproducible random seeds.
  • Live reactive preview of formatted JSONL records.
  • Machine-readable manifest handshake (mlx_manifest.json).
  • Supports Parquet, Arrow, Hugging Face save_to_disk, JSONL, JSON, CSV, TSV, SQLite, and WebDataset.

• Modern AI Agent & MCP Integration:

  • Full Model Context Protocol (MCP) server over stdio for Claude Desktop, Cursor, Antigravity, and Zed.
  • Native macOS Cocoa Finder file pickers and external macOS Terminal.app spawner.

Inspired by Norton Commander, with high-contrast, productive keybindings

📦 Quick Start

1. Installation

Install via pip from PyPI:

pip install mlx_commander

# Or install with all optional extras (Parquet, Arrow, DuckDB, Lance, MCP):
pip install "mlx_commander[all]"

Or run instantly without installation using uvx:

uvx mlx_commander

Or install globally as a standalone tool via uv:

uv tool install mlx_commander

2. Launching Modes

MLX Commander features a 3-mode switcher (F2 inside the TUI or via command-line flags):

# 1. Launch directly into Mode 2: Fine-Tuning Single Run
mlx_commander --lora

# 2. Launch directly into Mode 3: Fine-Tuning Multi-Run (Hyperparameter Sweeps)
mlx_commander --multi-run

# 3. Launch directly into Mode 1: Dataset Converter (Default)
mlx_commander

# 4. Execute all queued runs sequentially in an external window:
mlx_commander --run-queue ./mlx_runs

3. Headless Scripting & Automation

You can bypass the TUI entirely for automated pipelines and remote scripts:

Headless Dataset Conversion:

mlx_commander \
  --dataset /path/to/my_hf_dataset.parquet \
  --format prompt_completion \
  --prompt-col question \
  --completion-col answer \
  --train 80 --valid 10 --test 10 \
  --output ./mlx_data

Headless Sequential Queue Execution:

python3 -m mlx_commander --run-queue ./mlx_runs

🎛️ Mode 2: Fine-Tuning Single Run Orchestration

Press [F2] inside the TUI or pass --lora from the command line to enter Fine-Tuning Single Run Mode.

mlx_commander --lora

1. Dual-Panel Setup

• Top Left Panel (Model & Dataset Setup):

  • Base Model Picker: Quick-select from curated 4-bit Apple MLX community models (Llama-3.2-3B, Llama-3.1-8B, Qwen2.5-7B, Mistral-7B, Phi-3.5-mini, etc.) or input any custom Hugging Face model repository or local weights path.
  • Dataset Directory: Auto-synced from Mode 1 conversion output or selected via local path / native macOS Finder picker.
  • Method: Select lora, dora (Weight-Decomposed Low-Rank Adaptation), or full.
  • Optimizer: Pick adamw or adam.
  • Run Name: Custom label or auto-generated descriptive run signature.

• Top Right Panel (Explicit Hyperparameters & Resource Estimators):

  • 22 Explicit Parameters: All fine-tuning knobs exposed with tested defaults: iters, batch_size, gradient_accumulation_steps, learning_rate, lora_rank, lora_alpha, lora_dropout, max_seq_length, num_layers, grad_checkpoint, mask_prompt, save_every, steps_per_eval, and adapter_path.
  • Unified RAM Safety Estimator: Detects your Apple Silicon chip and physical RAM via sysctl hw.memsize and computes peak memory consumption:
    • [SAFE] (<70% RAM): Ample headroom for macOS and display compositor.
    • [TIGHT] (70–85% RAM): Viable, but approaching memory pressure limits.
    • [OOM RISK] (>85% RAM): Proactively warns before training begins and suggests enabling grad_checkpoint or reducing batch size.
  • Implied Number of Epochs: Calculated in real time via: $$\text{Epochs} = \frac{\text{iters} \times \text{batch_size} \times \text{gradient_accumulation_steps}}{\text{total_train_records}}$$ Highlighted in dark red if $< 1.0$ to alert you that the model will not observe the complete dataset.
  • Wall-Clock Duration & Clock ETA: Estimates execution duration and completion time based on your chip's compute throughput.

2. Vision-Language & Multimodal Safeguards (mlx_vlm)

When training multimodal models (such as Gemma 4, Qwen2-VL, PaliGemma, or SmolVLM) via mlx-vlm, micro-batch sizes greater than 1 trigger attention mask shape broadcasting errors due to unpadded multimodal token masks.

MLX Commander automatically detects mlx_vlm models and applies mathematical equivalence: $$\text{batch_size} \leftarrow 1$$ $$\text{gradient_accumulation_steps} \leftarrow \text{batch_size} \times \text{gradient_accumulation_steps}$$

This preserves identical effective batch size and gradient step dynamics while preventing runtime crashes:

[MLX-VLM Safeguard] Detected mlx-vlm model with batch_size=4.
Auto-tweaked to batch_size=1 and gradient_accumulation_steps=4 (effective batch size: 4).

3. Live Validation Loss Milestones

When valid.jsonl is present in the dataset folder, MLX Commander automatically loads it into val_dataset and computes validation loss every steps_per_eval iterations, highlighting progress milestones in the terminal:

★ [Validation Loss Milestone] Iter 100: Val loss = 1.234 (val took 0.54s)

🎛️ Mode 3: Fine-Tuning Multi-Run (Hyperparameter Sweeps)

Press [F2] inside the TUI or pass --multi-run from the command line to switch to Fine-Tuning Multi-Run Mode.

mlx_commander --multi-run

1. Single-Line Multi-Value Fields with Clickable Targets

Configure multiple sweep conditions directly on a single line:

Learning Rate:    [ 1e-4 ]  [ 2e-4 ]  [ 5e-4 ]
LoRA Rank (r):    [ 4 ]  [ 8 ]  [ 16 ]
Grad Accum Steps: [ 1 ]  [ 2 ]
Mask Prompt:      [ True ]  [ False ]

The rightmost bracketed box represents the active target. Press Enter to open a modal dialog to append or edit values as a comma-separated list (e.g. 1e-4, 2e-4, 5e-4).

2. Visual Cartesian Sweep Grid

The lower panel renders an interactive Cartesian product grid showing parameter columns, vertically stacked condition cells with dividers, centered ✖ multiplication symbols, and the total scheduled run count (e.g. $3 \times 3 \times 2 \times 2 = 36 \text{ runs}$).

Aggregated sweep estimates update reactively:

• Peak Unified RAM: Maximum RAM footprint across all sweep conditions.
• Total Duration: Sum of estimated wall-clock durations for all scheduled jobs.
• Min Implied Epochs: Minimum implied epochs across conditions.

3. Sequential FIFO Queue Execution

Press [F5 Run Queue]:

• Spawns a dedicated macOS Terminal.app window running mlx_commander --run-queue mlx_runs.
• Stdout, training loss, and throughput stream live.
• You can safely close the main MLX Commander TUI while background training proceeds undisturbed.

📊 Generative Test Set Evaluation Engine (experimental)

Standard mlx_lm.lora --test only computes cross-entropy loss and perplexity via teacher forcing—it never prompts the model to generate text.

MLX Commander features a built-in Generative Evaluation Engine tagged as (experimental). When enabled (Run evals on test set: [ Yes ] or run_eval=True), the fine-tuned model loads upon training completion, generates answers token-by-token on test.jsonl, and deterministically evaluates completions against reference targets without external scripts.

1. Physics-Grounded Throughput Engine & Dynamic ETA

Autoregressive token decoding is strictly memory-bandwidth bound: $$\text{Generation Speed (tok/s)} \approx \frac{\text{Unified Memory Bandwidth (GB/s)}}{\text{Active Model Weight Footprint (GB)}} \times 0.65$$

Throughput scales inversely with model parameter count and precision:

• 1B–3B 4-bit model (~0.7–2.0 GB): 150–350 tok/s (~35s for 50 samples).
• 8B 4-bit model (~5.2 GB): 40–60 tok/s (~1m 20s for 50 samples).
• 70B 4-bit model (~45.5 GB): 5–8 tok/s (~10m 30s for 50 samples).
• 8B fp16 unquantized (~16.0 GB): 14–18 tok/s (~3m 45s for 50 samples).

Pre-Evaluation Header:

Before inference starts, the runner prints explicit targets and estimated duration:

[MLX Commander] Starting Generative Evaluation on test set:
  • Target Checkpoint: adapters/01_run/adapters.safetensors (Final trained adapter)
  • Base Model:        mlx-community/Llama-3.2-3B-Instruct-4bit [Engine: mlx_lm]
  • Test Dataset:      dataset/test.jsonl (Full split: 50 samples)
  • Evaluation Passes: 2 passes (Baseline + Fine-Tuned = 100 generations)
  • Estimated Time:    ~35s (model: ~3B 4-bit [~2.0 GB], est. speed: ~150 tok/s on Apple M5 Max)

Real-Time Milestone Reporting:

During generation, live terminal updates display sample progress, speed, and countdown ETA:

  • [2/2 Fine-Tuned Adapter] Sample 25/50 (50%) | 148.4 tok/s | ETA: 18s

2. Evaluated Metrics

1. Exact Match (Strict & Normalized):
   • Strict: Character-for-character equality (gen == golden).
   • Normalized: SQuAD-standard matching stripping whitespace, punctuation, and English articles (a, an, the).
2. Substring Contains Match:
   • Verifies whether the golden completion appears inside the generated text.
3. Word-Level Precision, Recall, and F1 Score:
   • Evaluates factual overlap and gives fair partial credit when the model responds in full sentences.
4. Generation Speed & Efficiency:
   • Reports sustained tokens per second (TPS) and sample latency.

3. The "Matrix of Wrong Answers"

1. Categorical Confusion Matrix (for classification tasks with $\le 15$ classes):
   • Computes an [Actual] × [Predicted] confusion matrix with per-class recall and overall accuracy.
2. 2×2 Model Migration & Regression Matrix (comparing Pre-Trained Baseline vs Post-Tuning LoRA):

                        POST-TUNING (LoRA)
                     CORRECT           WRONG
                ┌────────────────┬────────────────┐
      CORRECT   │   PRESERVED    │   REGRESSED    │  ◄ Catastrophic forgetting!
BASELINE        ├────────────────┼────────────────┤
      WRONG     │     FIXED      │   PERSISTENT   │  ◄ Where LoRA healed the model!
                └────────────────┴────────────────┘

• FIXED: Baseline failed, but LoRA answered correctly (healed!).
• REGRESSED: Baseline answered correctly, but LoRA failed (catastrophic forgetting).
• PRESERVED: Both models answered correctly.
• PERSISTENT_FAIL: Hard samples failed by both models.

4. 3-Tier Storage & Offline Dashboard

Evaluation results are organized hierarchically:

models/Llama-3.2-3B/adapters/
├── eval_leaderboard.csv                    <-- TIER 1: Master CSV (open in Numbers/Excel)
├── eval_comparison.html                   <-- TIER 3: Standalone interactive dashboard
│
└── 01_lora_r16_a32_lr1e-4_b4_i1000/
    ├── adapters.safetensors
    ├── eval_summary.json                  <-- TIER 2: Run metadata & matrix stats
    └── eval_predictions.jsonl             <-- TIER 2: Row-by-row prompts, answers, & diffs

• Tier 1 (eval_leaderboard.csv): Appends a row for every completed run with F1, Exact Match %, regression counts, and speed. Open directly in Apple Numbers, Excel, or Google Sheets to rank sweep runs.
• Tier 2 (eval_summary.json & eval_predictions.jsonl): Saved inside each run's adapter folder with full prompts, outputs, and transition flags.
• Tier 3 (eval_comparison.html): Standalone offline HTML dashboard with metric scorecards, sortable leaderboards, and a Sample Explorer with filter buttons ([ All ], [ ⚠️ Regressed Only ], [ ❇️ Fixed Only ], [ Preserved ], [ Persistent Fail ]).
• Weights & Biases (W&B): Uploads interactive wandb.Table prediction diffs and confusion heatmaps.

🔄 Mode 1: Dataset Preparation & Ingestion

Press [F2] to switch to Dataset Converter Mode (the default startup view).

1. Supported MLX Formats

1. prompt_completion: Q&A, instruction pairs, or query/code.
2. chat: Multi-turn dialogue ({"messages": [...]}). Supports OpenAI / ShareGPT formats or separate role columns.
3. text: Raw causal language modeling / pre-training ({"text": "..."}).
4. dpo: Direct Preference Optimization ({"prompt": "...", "chosen": "...", "rejected": "..."}).

2. Multi-Column Concatenation & Multi-File Merging

• Tap Space on original dataset columns to combine multiple fields (e.g. instruction + input) joined by \n\n.
• Select multiple dataset files simultaneously (e.g. pre-split train.jsonl and test.jsonl). Verifies column schemas match and re-splits with reproducible random seeds.

3. Machine-Readable Manifest Handshake (mlx_manifest.json)

Every conversion emits a structured JSON manifest containing record counts, file sizes, and copy-paste fine-tuning commands:

{
  "status": "success",
  "format": "prompt_completion",
  "source_path": "/path/to/source.parquet",
  "output_dir": "/path/to/mlx_dataset",
  "files": {
    "train": {"path": "/path/to/mlx_dataset/train.jsonl", "records": 8000},
    "valid": {"path": "/path/to/mlx_dataset/valid.jsonl", "records": 1000},
    "test": {"path": "/path/to/mlx_dataset/test.jsonl", "records": 1000}
  },
  "total_records": 10000,
  "seed_used": 42,
  "mlx_lora_command": "mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit --train --data /path/to/mlx_dataset --mask-prompt --iters 600 --batch-size 4"
}

🤖 AI Agent Integration & MCP Server

MLX Commander includes a native Model Context Protocol (MCP) server over stdio designed for Antigravity, Claude Desktop, Cursor, Zed, and Cline.

1. Configuration

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "mlx_commander": {
      "command": "python3",
      "args": ["-m", "mlx_commander", "--mcp"]
    }
  }
}

Cursor (.cursor/mcp.json):

{
  "mcpServers": {
    "mlx_commander": {
      "command": "python3",
      "args": ["-m", "mlx_commander", "--mcp"]
    }
  }
}

2. Exposed MCP Tools

3. Exposed MCP Prompts

• prepare_dataset_for_mlx: Guides agents through inspecting schema, selecting format, and converting datasets.
• orchestrate_fine_tuning: Instructs agents on memory estimation, VLM safeguards, sweep queueing, sequential execution, and test evaluations.

🖥️ Command Line Reference

usage: mlx_commander [-h] [-v] [-d DATASET [DATASET ...]]
                     [-f {text,chat,prompt_completion,dpo}]
                     [-o OUTPUT] [--train TRAIN] [--valid VALID] [--test TEST]
                     [--seed SEED] [--keep-splits] [--mapping MAPPING]
                     [--text-col TEXT_COL] [--text-template TEXT_TEMPLATE]
                     [--prompt-col PROMPT_COL] [--completion-col COMPLETION_COL]
                     [--messages-col MESSAGES_COL] [--user-col USER_COL]
                     [--assistant-col ASSISTANT_COL] [--system-col SYSTEM_COL]
                     [--chosen-col CHOSEN_COL] [--rejected-col REJECTED_COL]
                     [--manifest-file MANIFEST_FILE] [--prefill-state PREFILL_STATE]
                     [--spawn-terminal] [--lora] [--multi-run]
                     [--run-queue [DIR]] [--mcp] [--tui] [--no-tui] [--wizard]

Key Flags:

🧪 Running Unit Tests

Run the full test suite with Python's standard library unittest:

python3 -m unittest discover -s tests

📜 License

MIT License. Designed with ❤️ for Apple Silicon and the open-source MLX ecosystem.