OPset for vLLM-HUST

OPset is a pluggable operator-optimization collection for the vLLM-HUST Ascend backend, delivering kernel replacements, vertical fusion, epilogue/prologue fusion, and the performance evidence for each change.

OPset is distributed as an updateable vLLM-HUST Extension Bundle with independently gated optimizations. Version 0.3.1 contains:

• persistent-matmul-empty: replaces the redundant torch.zeros allocation in Ascend batch-invariant Triton linear_persistent with torch.empty.
• linear-swiglu-graph: fuses adjacent unquantized gate/up GEMM and SwiGLU in the Graph path. The validated decode/small-batch range (M <= 128) uses the fused Triton kernel; larger dynamic prefill shapes fail closed to the native GEMM plus npu_swiglu path.

Both optimizations are limited to the batch-invariant Graph/Triton path and do not change ordinary ACLNN MatMul or unrelated linear implementations. Existing NPU evidence covers supported dtypes, divisible and non-divisible shapes, and matched operator/end-to-end measurements. Unknown source structures are rejected rather than patched.

Architecture

The distribution exposes two entry points:

• vllm_hust.extension_bundles lets vllm-hust-ext discover, check and enable the 0.2-experimental bundle.
• vllm.general_plugins installs a lazy import hook in each vLLM process. The hook changes only enabled target modules and has no device or network side effects at import time.

Optimizations are selected with VLLM_HUST_OPERATOR_OPTIMIZATIONS. The bundle manager enables both shipped optimizations; either ID can also be selected independently.

Install and use

This distribution succeeds vllm-hust-operator-optimizations. Existing Python imports, extension IDs, environment variables, and the vllm-hust-operator-check command remain available. Upgrade an existing installation with:

pip uninstall -y vllm-hust-operator-optimizations
pip install vllm-hust-opset==0.3.1

The package records the fixed experiment line in compatibility/verified-line.json. This keeps current results reproducible without repeatedly rebuilding the environment; the line will be refreshed and revalidated periodically as the underlying stack advances.

export VLLM_SRC=/path/to/vllm
export VLLM_ASCEND_SRC=/path/to/vllm-ascend-hust
pip install vllm-hust-opset==0.3.1
vllm-hust-opset \
  --vllm-src "$VLLM_SRC" \
  --vllm-ascend-src "$VLLM_ASCEND_SRC"
vllm-hust-ext extension check org.vllm-hust.operator-optimizations
vllm-hust-ext extension enable org.vllm-hust.operator-optimizations
VLLM_BATCH_INVARIANT=1 vllm-hust-ext run -- vllm serve /path/to/model \
  --block-size 128

TRITON_ASCEND_SRC may additionally point to the locked Triton checkout. If it is absent, the checker verifies the installed triton-ascend==3.2.2 wheel; this is the mode used by the validated server environment.

Repository maintainers can build with uv build --no-sources --out-dir dist and use scripts/install_verified_line.sh to gate and install a local wheel.

If VLLM_PLUGINS is manually set, it must include both the Ascend platform plugin name and operator_optimizations; otherwise vLLM intentionally filters out this runtime entry point.

Add another optimization

1. Add an Optimization implementation under optimizations/ with a stable ID, target module and fail-closed apply() method.
2. Register it in registry.register_builtins().
3. Add its ID to the manifest environment value if it should be enabled by this bundle version, or publish a separately selectable bundle profile.
4. Add unit correctness tests and NPU shape/tail, operator-share and matched end-to-end evidence.
5. Increment _version.py and the manifest extension_version together.

The framework deliberately keeps every optimization independent so a future release can add, disable or reject one optimization without changing the others.

Validation

uv run --with pytest pytest
uv run --with ruff ruff check src tests
uv build --no-sources --out-dir dist

For 0.3.1, the linear-SwiGLU Graph path was validated with Qwen2.5-7B, ShareGPT, three matched baseline/candidate rounds and operator correctness coverage. Steady-state TPOT decreased by 3.82% and output-token throughput increased by 5.24%. Release checks remain extension list, inspect, check, and run --dry-run; full NPU validation is refreshed periodically or when the optimization changes.