vLLM-HUST Operator Optimizations

This package is an updateable vLLM-HUST Extension Bundle for independently gated operator optimizations. Version 0.1.1 contains one optimization:

• persistent-matmul-empty: replaces the redundant torch.zeros allocation in Ascend batch-invariant Triton linear_persistent with torch.empty.

The optimization is limited to the batch-invariant Triton path. It does not change ordinary ACLNN MatMul or other linear implementations. The persistent kernel writes every valid output element, including tail tiles. Existing NPU evidence covers FP16/BF16/FP32, divisible and non-divisible M/N/K shapes, and matched operator/end-to-end measurements. Unknown source structures are rejected rather than patched.

Architecture

The distribution exposes two entry points:

• vllm_hust.extension_bundles lets vllm-hust-ext discover, check and enable the 0.2-experimental bundle.
• vllm.general_plugins installs a lazy import hook in each vLLM process. The hook changes only enabled target modules and has no device or network side effects at import time.

Optimizations are selected with VLLM_HUST_OPERATOR_OPTIMIZATIONS. The bundle manager sets it to persistent-matmul-empty when this extension is enabled.

Install and use

The supported release line is intentionally narrow: Ascend 910B2, CANN 9.1.0-beta.3, ATB 9.1.T6.B010, Python 3.11, Torch 2.10.0, torch_npu 2.10.0.post4 and Triton-Ascend 3.2.2. Exact source commits are in compatibility/verified-line.json inside the package. Other combinations are unsupported until separately validated.

export VLLM_SRC=/path/to/vllm
export VLLM_ASCEND_SRC=/path/to/vllm-ascend-hust
pip install vllm-hust-operator-optimizations==0.1.1
vllm-hust-operator-check \
  --vllm-src "$VLLM_SRC" \
  --vllm-ascend-src "$VLLM_ASCEND_SRC"
vllm-hust-ext extension check org.vllm-hust.operator-optimizations
vllm-hust-ext extension enable org.vllm-hust.operator-optimizations
VLLM_BATCH_INVARIANT=1 vllm-hust-ext run -- vllm serve /path/to/model \
  --block-size 128

TRITON_ASCEND_SRC may additionally point to the locked Triton checkout. If it is absent, the checker verifies the installed triton-ascend==3.2.2 wheel; this is the mode used by the validated server environment.

Repository maintainers can build with uv build --no-sources --out-dir dist and use scripts/install_verified_line.sh to gate and install a local wheel.

If VLLM_PLUGINS is manually set, it must include both the Ascend platform plugin name and operator_optimizations; otherwise vLLM intentionally filters out this runtime entry point.

Add another optimization

1. Add an Optimization implementation under optimizations/ with a stable ID, target module and fail-closed apply() method.
2. Register it in registry.register_builtins().
3. Add its ID to the manifest environment value if it should be enabled by this bundle version, or publish a separately selectable bundle profile.
4. Add unit correctness tests and NPU shape/tail, operator-share and matched end-to-end evidence.
5. Increment _version.py and the manifest extension_version together.

The framework deliberately keeps every optimization independent so a future release can add, disable or reject one optimization without changing the others.

Validation

uv run --with pytest pytest
uv run --with ruff ruff check src tests
uv build --no-sources --out-dir dist

For 0.1.1, reuse the existing Qwen2.5-7B Graph correctness and performance evidence for the exact locked line. Re-run NPU validation only when that line or the optimization changes. Release checks remain extension list, inspect, check, and run --dry-run; the completed real service run is not repeated merely for packaging changes.