vLLM x DFloat11
📦 Installation
pip install vllm-df11
Dependencies:
- vLLM >= 0.9.0
- CUDA-compatible GPU (A100, H100, H200, RTX, etc.)
You do not need nvcc or a C/C++ compiler to install this package.
However, a CUDA-enabled GPU is required to use dfloat11 with vLLM.
🚀 Usage
Enable the plugin by setting the environment variable:
import os
os.environ["VLLM_PLUGINS"] = "df11"
from vllm.plugins import load_general_plugins
load_general_plugins()
from vllm import LLM, SamplingParams
df11_model_path = "/path/to/dfloat11/e.g./llama3.1-8b-it-df11"
llm = LLM(
model=df11_model_path,
load_format="df11",
dtype="bfloat16"
)
prompts = ["Explain Huffman coding and describe its applications."]
sampling_params = SamplingParams(
temperature=0.6,
top_p=.95,
max_tokens=64,
)
outputs = llm.generate(prompts, sampling_params=sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt}")
print(f"Generated: {generated_text}")
print(" 🌳🦖🗜️📦🌲 " * 5)
📚 Reference
If you use this plugin in your research or deployment, please cite our paper.
Metadata
Release files for vllm-df11 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
Total release size: 4.7 MB
Release files / vllm_df11-0.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | vllm_df11-0.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
e7930d2d26ad49dd4fe3c0852f54f2e8f31df0d49874d5d3cccc72bf26b4b1b3
|
|
BLAKE2b-256 checksum How to use checksums |
1b85a28fcf9c5182fc461f0f4c1198e68db3542739dd31d8d2e2e06980d3fbb7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.7
|
Release files / vllm_df11-0.0.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | vllm_df11-0.0.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
9fc7e75ad66218053ec6ccefc1d8a9c40e16985f537b9d60df75caef023c09d9
|
|
BLAKE2b-256 checksum How to use checksums |
898338cc72daf900c3fd57402e04cc670630a79d0405ea68769e895f13af2923
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.7
|
Release files / vllm_df11-0.0.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | vllm_df11-0.0.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
5245ee5972d17e24f8fc4c8fa442e2388d85a97d849598a09ab3237084a2f432
|
|
BLAKE2b-256 checksum How to use checksums |
032de0b79903d7015a11a6a33c45ba0ab29670b2974ca9f8a86758746cf3c93b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.7
|
Release files / vllm_df11-0.0.1-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | vllm_df11-0.0.1-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.9 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
830e0459d101d218634dcb959f301e5db3b4a6e6e62d1b2ec8a13d35ad9e82ab
|
|
BLAKE2b-256 checksum How to use checksums |
d8c1e11d483b084d1341eb9460dc75e5a7a1ba70340a8351be13f30a917edd93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.7
|
Security for vllm-df11 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.