fastllm-turbo 1.0.0

Run AI models 2x faster on CPU with zero quality loss

Fast LLM

Run any AI model 2x faster on CPU — zero quality loss, zero training.

pip install fast-llm
fast-llm run Qwen/Qwen2.5-0.5B --quantize --benchmark

How it works

INT8 dynamic quantization converts model weights from 32-bit to 8-bit precision. This cuts memory bandwidth in half — the main bottleneck on CPU inference. The output is coherent English, indistinguishable from the original.

Mode	Speed	Output
FP32 (before)	1.18 tok/s	"The future of AI is in the hands of the people."
INT8 (after)	2.32 tok/s	"The future of AI is the future of the world."

Install

Linux / macOS

curl -sSf https://fast-llm.dev/install.sh | sh

Windows (PowerShell)

powershell -c "iex ((New-Object Net.WebClient).DownloadString('https://fast-llm.dev/install.ps1'))"

pip (all platforms)

pip install fast-llm

Usage

# Quantize + benchmark
fast-llm run Qwen/Qwen2.5-0.5B --quantize --benchmark

# Quantize only (faster, no benchmark)
fast-llm run microsoft/Phi-3-mini-4k-instruct --quantize

# Exact mode (same output, thread optimization only)
fast-llm run meta-llama/Llama-3.2-1B --exact --benchmark

Works with any HuggingFace causal language model: Qwen, Llama 3, Phi-3, Mistral, Gemma, DeepSeek, and more.

Requirements

• Python 3.8+
• PyTorch 2.0+ (CPU)
• 1–4 GB RAM per billion parameters

License

MIT