vllm-cpu-amxbf16 0.16.0

Easy, fast, and cheap LLM serving for everyone

Deprecation Notice: vllm-cpu-amxbf16 v0.16.0 is the last release of this variant package. Starting with v0.17.0, all CPU ISA variants are unified into a single package with automatic ISA detection at runtime. Migrate to: pip install vllm-cpu

See github.com/MekayelAnik/vllm-cpu for details.

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

About

vLLM is a fast and easy-to-use library for LLM inference and serving. This PyPl package has support for all the state of the art LLM inference instruction sets availble on most advanced CPUs: AVX512+VNNI+AVX512BF16+AMXBF16.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests
• Fast model execution with AVX512+VNNI+AVX512BF16+AMXBF16 on supported CPUs. Use this package ONLY IF your CPU has amxbf16 or newer instruction sets.
• Quantizations: GPTQ, AWQ, AutoRound, INT4, INT8, and FP8
• Optimized CPU kernels, including integration with FlashAttention and FlashInfer
• Speculative decoding
• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular Hugging Face models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Support for x86_64, PowerPC CPUs, Arm CPUs and Applie Scilicon (CPU inference). This package does not support any GPU inference. For GPU inference support use the official vLLM PypI
• Prefix caching support
• Multi-LoRA support

vLLM seamlessly supports most popular open-source models on HuggingFace, including:

• Transformer-like LLMs (e.g., Llama)
• Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
• Embedding Models (e.g., E5-Mistral)
• Multi-modal LLMs (e.g., LLaVA)

Find the full list of supported models here.

Important Notes

• Install this package on Linux envirenment only. For Windows you will have to use WSL2 or later
• This package has a Container.io (Docker/Podman etc.) compatible image in Docker Hub
• Apache Licence of main vLLM project
• GPL License of this CPU specific vLLM package
• For versions 0.8.5–0.12.0, use .post2 releases (e.g., pip install vllm-cpu-amxbf16==0.12.0.post2) — includes critical CPU platform detection fix

Platform Detection Fix (versions 0.8.5 - 0.12.0)

If you encounter RuntimeError: Failed to infer device type or see UnspecifiedPlatform warnings with versions 0.8.5 to 0.12.0, run this one-time fix after installation:

import os, sys, importlib.metadata as m
v = next((d.metadata['Version'] for d in m.distributions() if d.metadata['Name'].startswith('vllm-cpu')), None)
if v:
    p = next((p for p in sys.path if 'site-packages' in p and os.path.isdir(p)), None)
    if p:
        d = os.path.join(p, 'vllm-0.0.0.dist-info'); os.makedirs(d, exist_ok=True)
        open(os.path.join(d, 'METADATA'), 'w').write(f'Metadata-Version: 2.1\nName: vllm\nVersion: {v}+cpu\n')
        print(f'Fixed: vllm version set to {v}+cpu')

This creates a package alias so vLLM detects the CPU platform correctly. Only needed once per environment. Versions 0.8.5.post2+ and 0.12.0+ include this fix automatically.

Getting Started

Install vLLM with a single command:

pip install vllm-cpu-amxbf16 --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple

This installs vllm-cpu-amxbf16 with CPU-optimized PyTorch (no CUDA dependencies).

Alternative: Using uv (faster)

uv pip install vllm-cpu-amxbf16 --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple

Install uv on Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Docker Images

Pre-built Docker images are available on Docker Hub and GitHub Container Registry.

# Pull from Docker Hub
docker pull mekayelanik/vllm-cpu:amxbf16-latest

# Or from GitHub Container Registry
docker pull ghcr.io/mekayelanik/vllm-cpu:amxbf16-latest

# Run OpenAI-compatible API server
docker run -p 8000:8000 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  mekayelanik/vllm-cpu:amxbf16-latest \
  --model facebook/opt-125m

Available tags: amxbf16-latest, amxbf16-<version> (e.g., amxbf16-0.12.0)

Platforms: linux/amd64

vllm-cpu

This CPU specific vLLM has 5 optimized wheel packages from the upstream vLLM source code:

Each package is compiled with specific CPU instruction set flags for optimal inference performance.

Check Your CPU & Get Install Command

pkg=vllm-cpu
grep -q avx512f /proc/cpuinfo && pkg=vllm-cpu-avx512
grep -q avx512_vnni /proc/cpuinfo && pkg=vllm-cpu-avx512vnni
grep -q avx512_bf16 /proc/cpuinfo && pkg=vllm-cpu-avx512bf16
grep -q amx_bf16 /proc/cpuinfo && pkg=vllm-cpu-amxbf16
printf "\n\tRUN:\n\t\tuv pip install $pkg\n"

Example list of CPUs with their supported instruction sets

***Currently no AMD CPU support AMXBF16. AMD expected to include AMXBF16 support from AMD Zen 7 CPUs

Buy Me a Coffee

Your support encourages me to keep creating/supporting my open-source projects. If you found value in this project, you can buy me a coffee to keep me up all the sleepless nights.

• vLLM documentation to learn more.
• GitHub Repo for this package
• DockerHub
• List of Supported Models