Fast LLM

Run any AI model faster on CPU — same output, zero training.

pip install fastllm-turbo
fast-llm calibrate Qwen/Qwen2.5-0.5B

How it works

Most of a neural network's weights don't need full precision. We use a per-layer reconstruction error metric to identify which layers are sensitive to precision loss and which are robust. Sensitive layers stay FP32. Robust layers use INT8 dynamic quantization.

The result: a mixed-precision model that outputs the same tokens as the original, but runs faster because ~60% of computation uses 4× smaller weights.

Install

pip

pip install fastllm-turbo

Linux / macOS

curl -sSf https://fast-llm.dev/install.sh | sh

Windows (PowerShell)

powershell -c "iex ((New-Object Net.WebClient).DownloadString('https://fast-llm.dev/install.ps1'))"

Usage

# Step 1: Analyze which layers can be INT8
fast-llm calibrate Qwen/Qwen2.5-0.5B

# Step 2: Save optimized model
fast-llm save Qwen/Qwen2.5-0.5B

# Step 3: Run
fast-llm run Qwen/Qwen2.5-0.5B --benchmark

# Interactive chat
fast-llm chat Qwen/Qwen2.5-0.5B

# Desktop GUI
fast-llm gui

Works with any HuggingFace causal LM: Qwen, Llama 3, Phi-3, Mistral, Gemma, DeepSeek.

Requirements

• Python 3.8+
• PyTorch 2.0+ (CPU)
• 1–4 GB RAM per billion parameters

License

MIT