vllm-cpu-avx512bf16 0.16.0

Easy, fast, and cheap LLM serving for everyone

Deprecation Notice: vllm-cpu-avx512bf16 v0.16.0 is the last release of this variant package. Starting with v0.17.0, all CPU ISA variants are unified into a single package with automatic ISA detection at runtime. Migrate to: pip install vllm-cpu

See github.com/MekayelAnik/vllm-cpu for details.

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

About

vLLM is a fast and easy-to-use library for LLM inference and serving. This PyPl package has only supports AVX512+VNNI+AVX512BF16. No support for AMXBF16 is available in this package. CPU inference will have the above available instruction set accelerations.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests
• Fast model execution with AVX512+VNNI+AVX512BF16 on supported CPUs Use this package ONLY IF your CPU have avx512bf16 or newer instruction sets.
• Quantizations: GPTQ, AWQ, AutoRound, INT4, INT8, and FP8
• Optimized CPU kernels, including integration with FlashAttention and FlashInfer
• Speculative decoding
• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular Hugging Face models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Support for x86_64, PowerPC CPUs, Arm CPUs and Applie Scilicon (CPU inference). This package does not support any GPU inference. For GPU inference support use the official vLLM PypI
• Prefix caching support
• Multi-LoRA support

vLLM seamlessly supports most popular open-source models on HuggingFace, including:

• Transformer-like LLMs (e.g., Llama)
• Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
• Embedding Models (e.g., E5-Mistral)
• Multi-modal LLMs (e.g., LLaVA)

Find the full list of supported models here.

Important Notes

• Install this package on Linux envirenment only. For Windows you will have to use WSL2 or later
• This package has a Container.io (Docker/Podman etc.) compatible image in Docker Hub
• Apache Licence of main vLLM project
• GPL License of this CPU specific vLLM package
• For versions 0.8.5–0.12.0, use .post2 releases (e.g., pip install vllm-cpu-avx512bf16==0.12.0.post2) — includes critical CPU platform detection fix

Platform Detection Fix (versions 0.8.5 - 0.12.0)

If you encounter RuntimeError: Failed to infer device type or see UnspecifiedPlatform warnings with versions 0.8.5 to 0.12.0, run this one-time fix after installation:

import os, sys, importlib.metadata as m
v = next((d.metadata['Version'] for d in m.distributions() if d.metadata['Name'].startswith('vllm-cpu')), None)
if v:
    p = next((p for p in sys.path if 'site-packages' in p and os.path.isdir(p)), None)
    if p:
        d = os.path.join(p, 'vllm-0.0.0.dist-info'); os.makedirs(d, exist_ok=True)
        open(os.path.join(d, 'METADATA'), 'w').write(f'Metadata-Version: 2.1\nName: vllm\nVersion: {v}+cpu\n')
        print(f'Fixed: vllm version set to {v}+cpu')

This creates a package alias so vLLM detects the CPU platform correctly. Only needed once per environment. Versions 0.8.5.post2+ and 0.12.0+ include this fix automatically.

Getting Started

Install vLLM with a single command:

pip install vllm-cpu-avx512bf16 --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple

This installs vllm-cpu-avx512bf16 with CPU-optimized PyTorch (no CUDA dependencies).

Alternative: Using uv (faster)

uv pip install vllm-cpu-avx512bf16 --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple

Install uv on Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Docker Images

Pre-built Docker images are available on Docker Hub and GitHub Container Registry.

# Pull from Docker Hub
docker pull mekayelanik/vllm-cpu:avx512bf16-latest

# Or from GitHub Container Registry
docker pull ghcr.io/mekayelanik/vllm-cpu:avx512bf16-latest

# Run OpenAI-compatible API server
docker run -p 8000:8000 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  mekayelanik/vllm-cpu:avx512bf16-latest \
  --model facebook/opt-125m

Available tags: avx512bf16-latest, avx512bf16-<version> (e.g., avx512bf16-0.12.0)

Platforms: linux/amd64

vllm-cpu

This CPU specific vLLM has 5 optimized wheel packages from the upstream vLLM source code:

Each package is compiled with specific CPU instruction set flags for optimal inference performance.

Check Your CPU & Get Install Command

pkg=vllm-cpu
grep -q avx512f /proc/cpuinfo && pkg=vllm-cpu-avx512
grep -q avx512_vnni /proc/cpuinfo && pkg=vllm-cpu-avx512vnni
grep -q avx512_bf16 /proc/cpuinfo && pkg=vllm-cpu-avx512bf16
grep -q amx_bf16 /proc/cpuinfo && pkg=vllm-cpu-amxbf16
printf "\n\tRUN:\n\t\tuv pip install $pkg\n"

Example list of CPUs with their supported instruction sets

***Currently no AMD CPU support AMXBF16. AMD expected to include AMXBF16 support from AMD Zen 7 CPUs

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

• vLLM documentation to learn more.
• GitHub Repo for this package
• DockerHub
• List of Supported Models