Cortex

GPU-accelerated local LLMs on Apple Silicon, built for the terminal.

Cortex is a fast, native CLI for running and fine-tuning LLMs on Apple Silicon using MLX and Metal. It automatically detects chat templates, supports multiple model formats, and keeps your workflow inside the terminal.

Highlights

• Apple Silicon GPU acceleration via MLX (primary) and PyTorch MPS
• Multi-format model support: MLX, GGUF, SafeTensors, PyTorch, GPTQ, AWQ
• Built-in LoRA fine-tuning wizard
• Chat template auto-detection (ChatML, Llama, Alpaca, Gemma, Reasoning)
• Conversation history with autosave and export

Quick Start

pipx install cortex-llm
cortex

Inside Cortex:

• /download to fetch a model from HuggingFace
• /model to load or manage models
• /status to confirm GPU acceleration and current settings

Installation

Option A: pipx (recommended)

pipx install cortex-llm

Option B: from source

git clone https://github.com/faisalmumtaz/Cortex.git
cd Cortex
./install.sh

The installer checks Apple Silicon compatibility, creates a venv, installs dependencies from pyproject.toml, and sets up the cortex command.

Requirements

• Apple Silicon Mac (M1/M2/M3/M4)
• macOS 13.3+
• Python 3.11+
• 16GB+ unified memory (24GB+ recommended for larger models)
• Xcode Command Line Tools

Model Support

Cortex supports:

• MLX (recommended)
• GGUF (llama.cpp + Metal)
• SafeTensors
• PyTorch (Transformers + MPS)
• GPTQ / AWQ quantized models

Advanced Features

• Dynamic quantization fallback for PyTorch/SafeTensors models that do not fit GPU memory (INT8 preferred, INT4 fallback)
  • docs/dynamic-quantization.md
• MLX conversion with quantization recipes (4/5/8-bit, mixed precision) for speed vs quality control
  • docs/mlx-acceleration.md
• LoRA fine-tuning wizard for local adapters (/finetune)
  • docs/fine-tuning.md
• Template registry and auto-detection for chat formatting (ChatML, Llama, Alpaca, Gemma, Reasoning)
  • docs/template-registry.md
• Inference engine details and backend behavior
  • docs/inference-engine.md
• Tooling (experimental, WIP) for repo-scoped read/search and optional file edits with explicit confirmation
  • docs/cli.md

Important (Work in Progress): Tooling is actively evolving and should be considered experimental. Behavior, output format, and available actions may change; tool calls can fail; and UI presentation may be adjusted. Use tooling on non-critical work first, and always review any proposed file changes before approving them.

Configuration

Cortex reads config.yaml from the current working directory. For tuning GPU memory limits, quantization defaults, and inference parameters, see:

• docs/configuration.md

Documentation

Start here:

• docs/installation.md
• docs/cli.md
• docs/model-management.md
• docs/troubleshooting.md

Advanced topics:

• docs/mlx-acceleration.md
• docs/inference-engine.md
• docs/dynamic-quantization.md
• docs/template-registry.md
• docs/fine-tuning.md
• docs/development.md

Contributing

Contributions are welcome. See docs/development.md for setup and workflow.

License

MIT License. See LICENSE.

Note: Cortex requires Apple Silicon. Intel Macs are not supported.