Ascend Quant Toolkit and W8A8 Runtime Extension
This repository contains two separately packaged components:
- Ascend Quant Toolkit: offline calibration, ModelSlim conversion, contract export and evaluation evidence.
- vllm-ascend-quant-ext: read-only W8A8 runtime plugin loaded through
vLLM's native
vllm.general_pluginsinterface and registered with the vLLM-HUST Extension Manager as a static Manifest 0.2 bundle.
The Toolkit is not installed in the serving process. The Runtime Extension contains no dataset, calibration or model-conversion lifecycle. Model-weight quantization is also independent of Adaptive Quantized KV.
Frozen runtime baseline
The current alpha targets the frozen v1 release baseline and its closest
documented Ascend platform snapshot:
| Component | Supported revision/version |
|---|---|
| vLLM-HUST | ref v1, commit f18cf803c5f63625e2c71253ddaf8b0bad0bad1a |
| vLLM-Ascend-HUST | commit 74f0c0a272376412b51e1c1864803d5f3a0f1b5f |
| torch | 2.13.0 |
| torch-npu | 2.13.0rc1 |
| CANN | >=9.1,<9.2 |
The plugin intentionally fails closed for a different vLLM/vLLM-Ascend Git revision, even if its numeric package version appears compatible.
Offline quantization
Install the released distribution, or work from a source checkout:
python -m pip install ascend-quant-toolkit==0.4.2
# or, from a source checkout
git clone https://github.com/jxd1111/Ascend-LLM-quant.git
cd Ascend-LLM-quant
python -m pip install -e . --no-deps --no-build-isolation
ascend-quant-toolkit doctor --path /path/to/output-volume
ascend-quant-toolkit plan \
--recipe qwen25-w8a8 \
--model /path/to/Qwen2.5-14B-Instruct \
--output /path/to/Qwen2.5-14B-Instruct-w8a8 \
--device npu:7
Replace plan with quantize to execute conversion. A successful export
writes the provenance, ModelSlim description and closed runtime contract. The
contract binds config, indexes, weight shards and evidence files by size and
SHA-256. It is validation metadata; it does not rewrite weight tensors.
Runtime plugin
Install the independent distribution:
python -m pip install vllm-ascend-quant-ext==0.5.0
vllm-ascend-quant-ext check --model /path/to/w8a8-model
Enable it for the next vLLM process:
export VLLM_ASCEND_QUANT_EXT_ENABLE=1
export VLLM_ASCEND_QUANT_EXT_ARTIFACT=/path/to/w8a8-model
vllm-hust serve /path/to/w8a8-model \
--host 0.0.0.0 \
--port 18000
Leave VLLM_PLUGINS unset unless the deployment already maintains a complete
allowlist for every required Ascend platform and general plugin. The
extension-owned enable flag is the activation gate.
The installed wheel exposes exactly two entry points:
vllm.general_plugins/vllm_ascend_quant
vllm_hust.extension_bundles/org.vllm-hust.ascend-quant-runtime
The first is the runtime activation hook vLLM loads in every process. The
second is a static Manifest 0.2 locator: the vLLM-HUST Extension Manager
resolves it from distribution metadata and validates
manifests/vllm-hust-extension-v0.2.json before importing any implementation
module. Its registration name is identical to the manifest extension_id, and
no new vllm.* entry-point namespace is claimed.
The extension does not modify vLLM or vLLM-Ascend source. The plugin validates
the artifact and frozen host first, then registers the namespaced
ASCEND_QUANT_W8A8 linear scheme by delegating to the host's selected W8A8
implementation.
Extension Manager integration
When the Extension Manager tooling is available, discovery, configuration and enable intent belong to it while vLLM keeps the process lifecycle:
python -m pip install 'vllm-hust-ext @ git+https://github.com/vLLM-HUST/extension-manager.git'
vllm-hust-ext extension list
vllm-hust-ext extension validate org.vllm-hust.ascend-quant-runtime
vllm-hust-ext extension enable org.vllm-hust.ascend-quant-runtime
vllm-hust-ext run --dry-run -- vllm serve /path/to/w8a8-model
Installation alone has no runtime effect. The callback is idempotent and default-off. Unknown contract fields, incompatible versions/revisions, missing files, tensor mismatches and registration collisions abort activation.
Online W8A8 replay
The runtime path can replay the smoothed W8A8 factors on the BF16 checkpoint instead of loading the converted artifact. Generate the serving view from the BF16 source and the fitted attachment:
ascend-quant-toolkit build-online-view \
--model /path/to/Qwen2.5-14B-Instruct \
--attachment /path/to/attachment \
--output /path/to/online-view
The view symlinks the BF16 checkpoint, writes the per-linear
quant_model_description.json that makes the host select
ASCEND_QUANT_W8A8_ONLINE, and records the attachment it resolves its factors
from. No weight is copied or rewritten. Generation fails closed when the
attachment and the checkpoint disagree: a linear without a factor would stay
BF16 next to replayed weights, and a factor for an absent module would abort the
load. An attachment that did not pass every replay check is admitted only with
--allow-unverified.
Serve the view with the attachment selected:
export VLLM_ASCEND_QUANT_EXT_ENABLE=1
export VLLM_ASCEND_QUANT_EXT_SMOOTHING=/path/to/attachment
vllm-hust serve /path/to/online-view \
--host 0.0.0.0 \
--port 18001
The attachment has to pass the producer's verification: the runtime refuses an
attachment whose verified flag is false, so a fit that only clears a wider
tolerance needs the widened tolerance at build-smoothing time rather than a
weakened runtime check.
VLLM_ASCEND_QUANT_EXT_ARTIFACT stays unset: the description, not an
environment variable, decides the scheme of every linear. Passing
--online-view <view> --smoothing <attachment> to tools/npu_e2e.py adds this
group to the frozen-host gate and requires the online answer to match the
offline artifact's.
Disable, recover and uninstall
Stop the current vLLM process and start a new one after clearing selection:
unset VLLM_ASCEND_QUANT_EXT_ENABLE
unset VLLM_ASCEND_QUANT_EXT_ARTIFACT
python -m pip uninstall -y vllm-ascend-quant-ext
The extension never modifies model files. If the active model description
uses ASCEND_QUANT_W8A8, restoring the native W8A8_MIX description is an
explicit offline operation:
ascend-quant-toolkit restore-modelslim \
--model /path/to/model \
--recipe qwen25-w8a8
Documentation
- Architecture
- Current vLLM-Ascend quantization flow
- Runtime plugin design
- Artifact contract
- Acceptance matrix
- W8A8 evidence
- Runtime packaging and release
- Frozen v1 NPU E2E evidence
- Release checklist
Historical validation records describe the exact behavior of their published version and are not retroactively rewritten when integration policy changes.
Metadata
Release files for ascend-quant-toolkit 0.4.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ascend_quant_toolkit-0.4.2.tar.gz | 78.1 kB | With provenance | Details for ascend_quant_toolkit-0.4.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ascend_quant_toolkit-0.4.2-py3-none-any.whl | Python 3 | none | any | Details for ascend_quant_toolkit-0.4.2-py3-none-any.whl |
Total release size: 119.1 kB
Release files / ascend_quant_toolkit-0.4.2.tar.gz
| Download URL | ascend_quant_toolkit-0.4.2.tar.gz |
|---|---|
| Size | 78.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5ff91e5343a8a3faaf687cac690ab7b90d4afa96a209514a3224765899548781
|
|
BLAKE2b-256 checksum How to use checksums |
acdd2142cd7e098877540cd50276e83db026048d2173e253a8742cab6f063949
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.
Transparency logRelease files / ascend_quant_toolkit-0.4.2-py3-none-any.whl
| Download URL | ascend_quant_toolkit-0.4.2-py3-none-any.whl |
|---|---|
| Size | 41.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
57964380f85275e5e7d32d14bc6df94c020d6be70bfe47da0dcc430d946d2470
|
|
BLAKE2b-256 checksum How to use checksums |
92b8412c6925bba10f90e042d14bbd41bad19bb83d4470123ae2e5fb0b80e1f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.
Transparency logSecurity for ascend-quant-toolkit 0.4.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for ascend_quant_toolkit-0.4.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.