vllm-cpu-avx512vnni 0.8.5

vLLM CPU inference engine (AVX512 + VNNI optimized)

Easy, fast, and cheap LLM serving for everyone

---

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

---

About

vLLM is a fast and easy-to-use library for LLM inference and serving. This PyPl package has VNNI (AVX512+VNNI) inference built in on supported CPUs.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests
• Fast model execution with VNNI on supported CPUs Use this package ONLY IF your CPU have avx512vnni or newer instruction sets
• Quantizations: GPTQ, AWQ, AutoRound, INT4, INT8, and FP8
• Optimized CPU kernels, including integration with FlashAttention and FlashInfer
• Speculative decoding
• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular Hugging Face models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Support for x86_64, PowerPC CPUs, Arm CPUs and Applie Scilicon (CPU inference). This package does not support any GPU inference. For GPU inference support use the official vLLM PypI
• Prefix caching support
• Multi-LoRA support

vLLM seamlessly supports most popular open-source models on HuggingFace, including:

• Transformer-like LLMs (e.g., Llama)
• Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
• Embedding Models (e.g., E5-Mistral)
• Multi-modal LLMs (e.g., LLaVA)

Find the full list of supported models here.

Importnt Notes

• Install this package on Linux envirenment only. For Windows you will have to use WSL2 or later
• This package has a Container.io (Docker/Podman etc.) compatible image in Docker Hub
• Apache Licence of main vLLM project
• GPL License of this CPU specific vLLM package

Getting Started

Install vLLM with pip or uv

mkdir -p /path/to/vllm
cd /path/to/vllm
uv venv
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
uv pip install vllm-cpu-ax512vnni

Install uv on Linux environment using CURL:

curl -LsSf https://astral.sh/uv/install.sh | sh

or using WGET

wget -qO- https://astral.sh/uv/install.sh | sh

if you wanna install a specific version of uv

curl -LsSf https://astral.sh/uv/0.9.11/install.sh | sh

vllm-cpu

This CPU specific vLLM has 5 optimized wheel packages from the upstream vLLM source code:

Package	Optimizations	Target CPUs
vllm-cpu	Baseline (no AVX512)	All x86_64 and ARM64 CPUs
vllm-cpu-avx512	AVX512	Intel Skylake-X and newer
vllm-cpu-avx512vnni	AVX512 + VNNI	Intel Cascade Lake and newer
vllm-cpu-avx512bf16	AVX512 + VNNI + BF16	Intel Cooper Lake and newer
vllm-cpu-amxbf16	AVX512 + VNNI + BF16 + AMX	Intel Sapphire Rapids (4th gen Xeon+)

Each package is compiled with specific CPU instruction set flags for optimal inference performance.

Check available CPU instruction sets

lscpu | grep -i flags

Example list of CPUs with their supported instruction sets

CPU Architecture (Intel/AMD)	AVX2	AVX-512 F (Base)	VNNI (INT8)	BF16 (BFloat16) (via AVX-512)	AMX-BF16 (via Tile Unit)
Intel 4th Gen / AMD Ryzen Zen2 & Newer	Yes	No	No	No	No
Intel Skylake-SP / Skylake-X / AMD Zen 4 & Newer	Yes	Yes	No	No	No
Intel Cooper Lake (3rd Gen Xeon) / AMD Zen 4 (EPYC) / Ryzen Zen5 & Newer	Yes	Yes	Yes	Yes	No
Intel Sapphire Rapids (4th Gen Xeon) & Newer	Yes	Yes	Yes	Yes	Yes

***Currently no AMD CPU support AMXBF16. AMD expected to include AMXBF16 support from AMD Zen 7 CPUs

---

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

---

• vLLM documentation to learn more.
• GitHub Repo for this package
• DockerHub
• List of Supported Models