vLLM GGUF Quantization Plugin

This plugin provides out-of-tree GGUF quantization support for vLLM after in-tree support deprecation (vllm-project/vllm#39583).

Installation

Prerequisites

CUDA toolkit or ROCm toolkit

We recommend uv for package management. If you don't have it installed:

curl -LsSf https://astral.sh/uv/install.sh | sh

From Source

Clone this repository:

git clone https://github.com/vllm-project/vllm-gguf-plugin
cd vllm-gguf-plugin

If vLLM is not already installed, install it first:
```
uv pip install vllm --torch-backend=auto
```
Build and install the plugin against the PyTorch installation used by vLLM:
```
uv pip install -e . --no-build-isolation
```
Disabling build isolation ensures that the CUDA extension is compiled against the same PyTorch installation used by vLLM at runtime.

Development

After completing the editable source installation above, install and run the development tooling:

uv pip install -e .[dev] --torch-backend=auto
pre-commit install
pre-commit run --all-files

The same hooks also run in GitHub Actions on every push and pull request.

Usage

vllm serve Qwen/Qwen3-0.6B-GGUF:Q8_0 --tokenizer Qwen/Qwen3-0.6B

Tested model coverage

The plugin uses vLLM's model implementations and a generic GGUF weight adapter, so model compatibility is broader than a fixed allowlist. The models below are covered by the repository's generation tests and are the best-known starting points:

Modality	Model family	Tested GGUF quantization
Text	Qwen 2.5	Q6_K
Text	Qwen 3	Q8_0
Text	Phi 3.5	IQ4_XS
Text	GPT-2	Q4_K_M
Text	StableLM	Q4_K_M
Text	Gemma 3	Q4_0
Text	OLMoE	Q4_0
Vision-language	Gemma 3	Q4_0 backbone with F16 projector
Image generation	Z-Image-Turbo	Q4_0
Image generation	FLUX.2-klein	Q8_0

Other vLLM-supported architectures may work when their GGUF tensor names map to the corresponding Hugging Face model. A model appearing in vLLM's general supported-model list does not by itself guarantee GGUF compatibility. When reporting an unsupported model, include the model repository, quantization, plugin and vLLM versions, and the complete weight-mapping error.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vllm_gguf_plugin-0.0.5.tar.gz (114.9 kB view details)

Uploaded Aug 10, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl (6.4 MB view details)

Uploaded Aug 10, 2026 CPython 3.10+manylinux: glibc 2.28+ x86-64

vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl (6.4 MB view details)

Uploaded Aug 10, 2026 CPython 3.10+manylinux: glibc 2.28+ ARM64

File details

Details for the file vllm_gguf_plugin-0.0.5.tar.gz.

File metadata

Download URL: vllm_gguf_plugin-0.0.5.tar.gz
Upload date: Aug 10, 2026
Size: 114.9 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vllm_gguf_plugin-0.0.5.tar.gz
Algorithm	Hash digest
SHA256	`4c786c5ee284bb616f3a72c54c423c08d7ea6247ef14108421520de4c923776f`
MD5	`7ca039a7708202c49b231e2bf5515335`
BLAKE2b-256	`ab0d608dd195cff255708a6b5edb931e9f705def22fefe78a253625ac1194e1c`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vllm_gguf_plugin-0.0.5.tar.gz:

Publisher: release.yml on vllm-project/vllm-gguf-plugin

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vllm_gguf_plugin-0.0.5.tar.gz
- Subject digest: 4c786c5ee284bb616f3a72c54c423c08d7ea6247ef14108421520de4c923776f
- Sigstore transparency entry: 2406063501
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: vllm-project/vllm-gguf-plugin@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Branch / Tag: refs/tags/v0.0.5
- Owner: https://github.com/vllm-project
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Trigger Event: release

File details

Details for the file vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl.

File metadata

Download URL: vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl
Upload date: Aug 10, 2026
Size: 6.4 MB
Tags: CPython 3.10+, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`e795779830ad8907cb0e51bf661dd7784ebaf244b0b324af07a8d3925b6a9739`
MD5	`4de3ce541150f127bce618fa1147d271`
BLAKE2b-256	`0b1f55b5023c20ef8c30c8099d713aa2be94783d7a67aa31f87963779d82f5c6`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl:

Publisher: release.yml on vllm-project/vllm-gguf-plugin

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl
- Subject digest: e795779830ad8907cb0e51bf661dd7784ebaf244b0b324af07a8d3925b6a9739
- Sigstore transparency entry: 2406063509
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: vllm-project/vllm-gguf-plugin@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Branch / Tag: refs/tags/v0.0.5
- Owner: https://github.com/vllm-project
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Trigger Event: release

File details

Details for the file vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl.

File metadata

Download URL: vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl
Upload date: Aug 10, 2026
Size: 6.4 MB
Tags: CPython 3.10+, manylinux: glibc 2.28+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl
Algorithm	Hash digest
SHA256	`36a1538d0ac6797c8b194c12d94eceed944fe945dbc2585830393cb4c3b92d5b`
MD5	`366f2251d95603bb6aba75307331714f`
BLAKE2b-256	`a1a6e99dbaa266bc0502f83d4a167fb655755b5df029334d2ffc34209195b240`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl:

Publisher: release.yml on vllm-project/vllm-gguf-plugin

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_aarch64.whl
- Subject digest: 36a1538d0ac6797c8b194c12d94eceed944fe945dbc2585830393cb4c3b92d5b
- Sigstore transparency entry: 2406063515
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: vllm-project/vllm-gguf-plugin@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Branch / Tag: refs/tags/v0.0.5
- Owner: https://github.com/vllm-project
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@d358f564fc8f470cddd7c141a149b4ebafafa01f
- Trigger Event: release