fastllm-turbo 2.0.0

ONA-guided mixed-precision: faster LLM with same output

Fast LLM

Run any AI model faster on CPU — same output, zero training.

pip install fastllm-turbo
fast-llm calibrate Qwen/Qwen2.5-0.5B

How it works

Most of a neural network's weights don't need full precision. We use ONA's reconstruction loss to identify which layers are sensitive to precision loss and which are robust. Sensitive layers stay FP32. Robust layers use INT8 dynamic quantization.

The result: mixed-precision model that outputs the same tokens as the original, but runs faster because ~60% of computation uses 4× smaller weights.

Method	Speed	Output matches original?
INT8 all layers	~2×	No (quality degrades on small models)
ONA mixed-precision	~1.6×	Yes
FP32 (original)	1×	—

Install

pip

pip install fastllm-turbo

Linux / macOS

curl -sSf https://fast-llm.dev/install.sh | sh

Windows (PowerShell)

powershell -c "iex ((New-Object Net.WebClient).DownloadString('https://fast-llm.dev/install.ps1'))"

Usage

# Step 1: Analyze which layers can be INT8
fast-llm calibrate Qwen/Qwen2.5-0.5B

# Step 2: Save optimized model
fast-llm save Qwen/Qwen2.5-0.5B

# Step 3: Run
fast-llm run Qwen/Qwen2.5-0.5B --benchmark

# Interactive chat
fast-llm chat Qwen/Qwen2.5-0.5B

# Desktop GUI
fast-llm gui

Works with any HuggingFace causal LM: Qwen, Llama 3, Phi-3, Mistral, Gemma, DeepSeek.

How ONA fits in

ONA (Omni Neural Architecture) is a neural network built from scratch in NumPy with per-neuron attention and forward-pass learning. Its reconstruction loss — originally designed for self-supervised learning without backprop — doubles as a precision saliency metric. We reused this metric to solve a completely different problem: deciding which layers of a transformer model can be safely quantized.

ONA reconstruction loss  →  layer sensitivity analysis  →  mixed-precision model

Requirements

• Python 3.8+
• PyTorch 2.0+ (CPU)
• 1–4 GB RAM per billion parameters

License

MIT