Skip to main content

Ascend Quant Toolkit and W8A8 Runtime Extension

This repository contains two separately packaged components:

  • Ascend Quant Toolkit: offline calibration, ModelSlim conversion, contract export and evaluation evidence.
  • vllm-ascend-quant-ext: read-only W8A8 runtime plugin loaded through vLLM's native vllm.general_plugins interface and registered with the vLLM-HUST Extension Manager as a static Manifest 0.2 bundle.

The Toolkit is not installed in the serving process. The Runtime Extension contains no dataset, calibration or model-conversion lifecycle. Model-weight quantization is also independent of Adaptive Quantized KV.

Frozen runtime baseline

The current alpha targets the frozen v1 release baseline and its closest documented Ascend platform snapshot:

Component Supported revision/version
vLLM-HUST ref v1, commit f18cf803c5f63625e2c71253ddaf8b0bad0bad1a
vLLM-Ascend-HUST commit 74f0c0a272376412b51e1c1864803d5f3a0f1b5f
torch 2.13.0
torch-npu 2.13.0rc1
CANN >=9.1,<9.2

The plugin intentionally fails closed for a different vLLM/vLLM-Ascend Git revision, even if its numeric package version appears compatible.

Offline quantization

Install the released distribution, or work from a source checkout:

python -m pip install ascend-quant-toolkit==0.4.2

# or, from a source checkout
git clone https://github.com/jxd1111/Ascend-LLM-quant.git
cd Ascend-LLM-quant
python -m pip install -e . --no-deps --no-build-isolation

ascend-quant-toolkit doctor --path /path/to/output-volume
ascend-quant-toolkit plan \
  --recipe qwen25-w8a8 \
  --model /path/to/Qwen2.5-14B-Instruct \
  --output /path/to/Qwen2.5-14B-Instruct-w8a8 \
  --device npu:7

Replace plan with quantize to execute conversion. A successful export writes the provenance, ModelSlim description and closed runtime contract. The contract binds config, indexes, weight shards and evidence files by size and SHA-256. It is validation metadata; it does not rewrite weight tensors.

Runtime plugin

Install the independent distribution:

python -m pip install vllm-ascend-quant-ext==0.5.0
vllm-ascend-quant-ext check --model /path/to/w8a8-model

Enable it for the next vLLM process:

export VLLM_ASCEND_QUANT_EXT_ENABLE=1
export VLLM_ASCEND_QUANT_EXT_ARTIFACT=/path/to/w8a8-model

vllm-hust serve /path/to/w8a8-model \
  --host 0.0.0.0 \
  --port 18000

Leave VLLM_PLUGINS unset unless the deployment already maintains a complete allowlist for every required Ascend platform and general plugin. The extension-owned enable flag is the activation gate.

The installed wheel exposes exactly two entry points:

vllm.general_plugins/vllm_ascend_quant
vllm_hust.extension_bundles/org.vllm-hust.ascend-quant-runtime

The first is the runtime activation hook vLLM loads in every process. The second is a static Manifest 0.2 locator: the vLLM-HUST Extension Manager resolves it from distribution metadata and validates manifests/vllm-hust-extension-v0.2.json before importing any implementation module. Its registration name is identical to the manifest extension_id, and no new vllm.* entry-point namespace is claimed.

The extension does not modify vLLM or vLLM-Ascend source. The plugin validates the artifact and frozen host first, then registers the namespaced ASCEND_QUANT_W8A8 linear scheme by delegating to the host's selected W8A8 implementation.

Extension Manager integration

When the Extension Manager tooling is available, discovery, configuration and enable intent belong to it while vLLM keeps the process lifecycle:

python -m pip install 'vllm-hust-ext @ git+https://github.com/vLLM-HUST/extension-manager.git'
vllm-hust-ext extension list
vllm-hust-ext extension validate org.vllm-hust.ascend-quant-runtime
vllm-hust-ext extension enable org.vllm-hust.ascend-quant-runtime
vllm-hust-ext run --dry-run -- vllm serve /path/to/w8a8-model

Installation alone has no runtime effect. The callback is idempotent and default-off. Unknown contract fields, incompatible versions/revisions, missing files, tensor mismatches and registration collisions abort activation.

Online W8A8 replay

The runtime path can replay the smoothed W8A8 factors on the BF16 checkpoint instead of loading the converted artifact. Generate the serving view from the BF16 source and the fitted attachment:

ascend-quant-toolkit build-online-view \
  --model /path/to/Qwen2.5-14B-Instruct \
  --attachment /path/to/attachment \
  --output /path/to/online-view

The view symlinks the BF16 checkpoint, writes the per-linear quant_model_description.json that makes the host select ASCEND_QUANT_W8A8_ONLINE, and records the attachment it resolves its factors from. No weight is copied or rewritten. Generation fails closed when the attachment and the checkpoint disagree: a linear without a factor would stay BF16 next to replayed weights, and a factor for an absent module would abort the load. An attachment that did not pass every replay check is admitted only with --allow-unverified.

Serve the view with the attachment selected:

export VLLM_ASCEND_QUANT_EXT_ENABLE=1
export VLLM_ASCEND_QUANT_EXT_SMOOTHING=/path/to/attachment

vllm-hust serve /path/to/online-view \
  --host 0.0.0.0 \
  --port 18001

The attachment has to pass the producer's verification: the runtime refuses an attachment whose verified flag is false, so a fit that only clears a wider tolerance needs the widened tolerance at build-smoothing time rather than a weakened runtime check.

VLLM_ASCEND_QUANT_EXT_ARTIFACT stays unset: the description, not an environment variable, decides the scheme of every linear. Passing --online-view <view> --smoothing <attachment> to tools/npu_e2e.py adds this group to the frozen-host gate and requires the online answer to match the offline artifact's.

Disable, recover and uninstall

Stop the current vLLM process and start a new one after clearing selection:

unset VLLM_ASCEND_QUANT_EXT_ENABLE
unset VLLM_ASCEND_QUANT_EXT_ARTIFACT
python -m pip uninstall -y vllm-ascend-quant-ext

The extension never modifies model files. If the active model description uses ASCEND_QUANT_W8A8, restoring the native W8A8_MIX description is an explicit offline operation:

ascend-quant-toolkit restore-modelslim \
  --model /path/to/model \
  --recipe qwen25-w8a8

Documentation

Historical validation records describe the exact behavior of their published version and are not retroactively rewritten when integration policy changes.

Metadata

Release files for ascend-quant-toolkit 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ascend-quant-toolkit 0.4.2
File Size Uploaded Provenance Actions
ascend_quant_toolkit-0.4.2.tar.gz 78.1 kB With provenance Details for ascend_quant_toolkit-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ascend-quant-toolkit 0.4.2
File Interpreter ABI Platform Actions
ascend_quant_toolkit-0.4.2-py3-none-any.whl Python 3 none any Details for ascend_quant_toolkit-0.4.2-py3-none-any.whl

Total release size: 119.1 kB

Release files / ascend_quant_toolkit-0.4.2.tar.gz

Download URL ascend_quant_toolkit-0.4.2.tar.gz
Size 78.1 kB
Tags Source
SHA-256 checksum
How to use checksums
5ff91e5343a8a3faaf687cac690ab7b90d4afa96a209514a3224765899548781
BLAKE2b-256 checksum
How to use checksums
acdd2142cd7e098877540cd50276e83db026048d2173e253a8742cab6f063949
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.

Transparency log

Release files / ascend_quant_toolkit-0.4.2-py3-none-any.whl

Download URL ascend_quant_toolkit-0.4.2-py3-none-any.whl
Size 41.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
57964380f85275e5e7d32d14bc6df94c020d6be70bfe47da0dcc430d946d2470
BLAKE2b-256 checksum
How to use checksums
92b8412c6925bba10f90e042d14bbd41bad19bb83d4470123ae2e5fb0b80e1f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.

Transparency log

Security for ascend-quant-toolkit 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ascend_quant_toolkit-0.4.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.2 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page