This release is a pre-release and may not be stable for production use.
Ascend INT8 KV Cache Plugin
面向 vllm-ascend-hust 的 INT8 KV-cache attention implementation 插件,
运行在由 vllm-hust 启动的推理进程中。仓库只提供一种量化方式:运行时
动态 per-channel INT8;它不是 CUDA、ROCm 或 CPU 可用的通用 vLLM 插件。
插件不读取模型 checkpoint 的 fa_quant_type,也不要求模型预量化。
唯一启用开关是 vLLM 命令行参数:
vllm serve MODEL --kv-cache-dtype int8
运行机制
安装 wheel 后,vLLM 通过 vllm.general_plugins 动态加载
bootstrap.register_plugins。入口为宿主 AscendAttentionBackend 安装
get_impl_cls 分派器:cache dtype 为 int8 时返回插件实现,其他 dtype
继续调用宿主原始分派逻辑。未传 --kv-cache-dtype int8 时不会执行 INT8
量化路径。
INT8 scale 在每层首次收到 K/V 时沿 token 维计算,K/V 分别使用动态对称 per-channel scale。decode 使用 Ascend fused attention 的在线 antiquant; prefill 和 chunked prefill 在需要时 gather 并反量化分页缓存。
实际类组合为:
AscendInt8KvAttentionImpl
= plugin AscendInt8AttentionBackendMixin
+ host AscendAttentionBackendImpl
当 context parallel 开启时 INT8 KV cache 会明确拒绝启动,与当前宿主限制一致。
安装与运行
conda activate vllm-hust-dev
python -m pip install -e .
VLLM_LOGGING_LEVEL=INFO vllm serve MODEL \
--kv-cache-dtype int8 \
--max-model-len 8192
不需要 VLLM_HUST_KV_METHODS,不需要修改 checkpoint。
检查动态插件入口:
python -c "from importlib.metadata import entry_points; print([e for e in entry_points(group='vllm.general_plugins') if e.name == 'vllm-ascend-int8-kv-cache'])"
Bundle manifest
wheel 内包含 Bundle v1 manifest:
vllm_ascend_quantized_kv_cache/manifests/vllm-hust-extension-v1.json。
其宿主声明为 provider=vllm、name=vllm-ascend。
需要静态准入的宿主可通过 VLLM_EXTENSION_MANIFESTS 显式传入该文件。
Manifest 的 implementation_ref 描述插件提供的 backend 组件;实际运行时
接入由 vllm.general_plugins 调用 install_int8_impl_dispatch() 完成。
验证
PYTHONPATH=src python -m pytest -q
python -m build
bash scripts/verify-wheel.sh dist/*.whl
设备执行仅支持 Ascend NPU。当前目标环境为 Ascend 910B +
vllm-hust-dev。
已验证环境
2026-09-16 完成了真实 NPU 端到端验证:
| 项目 | 已验证值 |
|---|---|
| 插件版本 | 0.2.0.dev0 |
| vLLM-HUST commit | 8a6655cf62 |
| vLLM-Ascend-HUST commit | f4f49832 |
| vLLM 运行时版本 | 0.23.1.post1.dev498+g802ead286.dirty |
| 设备 | Ascend 910B,单卡 |
| 模型 | Qwen2.5-14B-Instruct |
| 模型权重 | BF16 |
| KV cache | 动态 per-channel INT8 |
| 上下文长度 | 8192 |
| Prefill / decode | 通过 |
| ACL Graph capture / replay | 通过 |
| OpenAI Chat API | 3 次请求均返回 HTTP 200 |
该验证不代表已覆盖多卡、context parallel、所有模型或所有宿主版本。 正式发布前应使用目标 wheel 在每个声明支持的宿主版本上重复验证。
Metadata
Release files for vllm-ascend-quantized-kv-cache 0.2.0rc4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vllm_ascend_quantized_kv_cache-0.2.0rc4.tar.gz | 1.7 MB | Without provenance | Details for vllm_ascend_quantized_kv_cache-0.2.0rc4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vllm_ascend_quantized_kv_cache-0.2.0rc4-py3-none-any.whl | Python 3 | none | any | Details for vllm_ascend_quantized_kv_cache-0.2.0rc4-py3-none-any.whl |
Total release size: 1.8 MB
Release files / vllm_ascend_quantized_kv_cache-0.2.0rc4.tar.gz
| Download URL | vllm_ascend_quantized_kv_cache-0.2.0rc4.tar.gz |
|---|---|
| Size | 1.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f2651aeb4fb299980805200b19213eafee1ac7ee2a1132ea92d0a91d62812513
|
|
BLAKE2b-256 checksum How to use checksums |
b37caff6503b98724d781fe528d0d8d1f0b939459e828eb858c974fe8ff7bff8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.17 {"installer":{"name":"uv","version":"0.12.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / vllm_ascend_quantized_kv_cache-0.2.0rc4-py3-none-any.whl
| Download URL | vllm_ascend_quantized_kv_cache-0.2.0rc4-py3-none-any.whl |
|---|---|
| Size | 37.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
efbde24322f450b2df222ee6348342c215034d527e847663305bfd8dd112d514
|
|
BLAKE2b-256 checksum How to use checksums |
f5ca148f87da95494f941ccb051a453f0b4917057afdc044668b3183f2186705
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.17 {"installer":{"name":"uv","version":"0.12.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for vllm-ascend-quantized-kv-cache 0.2.0rc4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.