Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

xlite logo

Xlite轻量化推理运行时

介绍

xlite (GVirt前端):轻量级Transformer模型运行时,支持多样性算力协同,当前支持在昇腾硬件上高效运行。 xlite公开了Transformer运行所需的模型构图以及算子,所有算子基于昇腾AscendC/CCE开发。

背景与动机

在大模型推理场景中,传统的单流串行执行模式存在以下问题:

  • 核间负载不均:不同AICORE之间的任务分配不均衡,部分核心闲置
  • 资源浪费多:计算资源和传输资源利用率低,存在显著浪费
  • 执行时间长:Host CPU下发算子开销大,造成严重的host bond问题

xlite通过以下技术手段解决上述问题:

  • 多流并行:充分利用卡内资源,将单流串行改为多流并行执行
  • 核间负载均衡:核间负载均衡,提升资源利用率
  • CPU NPU协同:C++侧完全消除Python的GC、线程等干扰,简化Host tiling计算,去除小块内存申请释放及拷贝,消除Host bond

性能效果显著:

在GLM-4.7双机推理场景(40K输入、1K输出、prefix cache命中率约90%)下:

  • TPOT时延降低17%~30%
  • 吞吐提升13%~41%

详细性能数据参考 PR #7935。

软件架构

image

xlite已适配vllm_ascend,可通过xlite_graph_config配置快速使能xlite加速效果,使用方法参考官方指导文档。

xlite支持模型见模型列表。

快速开始

安装

# 安装vllm_ascend, 可参考https://github.com/vllm-project/vllm-ascend/blob/main/README.md
# 安装xlite
pip install xlite --extra-index https://download.pytorch.org/whl/cpu/

离线推理示例

import os
from vllm import LLM

# xlite默认支持decode-only模式, 可通过设置 "full_mode": True 使能full模式
model = LLM(model="path/to/Qwen3-32B", tensor_parallel_size=8, additional_config={"xlite_graph_config": {"enabled": True, "full_mode": True}})
outputs = model.generate("Hello, how are you?")

在线服务示例

vllm serve path/to/Qwen3-32B --tensor-parallel-size 8 --additional-config='{"xlite_graph_config": {"enabled": true, "full_mode": true}}'

开发指南

编译构建、容器镜像、源码安装等开发相关内容,请参考 开发指南

环境变量

xlite使用的环境变量及其配置方法,请参考 环境变量

目录结构

.
├── csrc/    - 轻量化运行时的核心代码
├── xlite/   - python代码,包括tools等
├── doc/     - 相关文档介绍
├── docker/  - 容器镜像Dockerfile
└── tests/   - 测试用例

致谢

本项目的核心算子初始版本和思路来自于华为终端小艺AI Infra团队的贡献,相关优化实现可参考论文:

《XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads》 [ASPLOS 2026]

Metadata

Release files for xlite 0.2.0rc2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for xlite 0.2.0rc2
File Size Uploaded Provenance Actions
xlite-0.2.0rc2.tar.gz 335.3 kB Without provenance Details for xlite-0.2.0rc2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for xlite 0.2.0rc2
File Actions
xlite-0.2.0rc2-cp312-cp312-manylinux2014_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ x86-64 Details for xlite-0.2.0rc2-cp312-cp312-manylinux2014_x86_64.whl
xlite-0.2.0rc2-cp312-cp312-manylinux2014_aarch64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ ARM64 Details for xlite-0.2.0rc2-cp312-cp312-manylinux2014_aarch64.whl
xlite-0.2.0rc2-cp311-cp311-manylinux2014_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details for xlite-0.2.0rc2-cp311-cp311-manylinux2014_x86_64.whl
xlite-0.2.0rc2-cp311-cp311-manylinux2014_aarch64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ ARM64 Details for xlite-0.2.0rc2-cp311-cp311-manylinux2014_aarch64.whl

Total release size: 4.9 MB

Release files / xlite-0.2.0rc2.tar.gz

Download URL xlite-0.2.0rc2.tar.gz
Size 335.3 kB
Tags Source
SHA-256 checksum
How to use checksums
fc483510f9f781e91f7b0259ee30f766b470d61c9f8556b24c78eac80de44390
BLAKE2b-256 checksum
How to use checksums
7aaa9d78caa63603c0e1a122fdbd3614ddacd9cd54b9e9d29b492658c1a77842
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Release files / xlite-0.2.0rc2-cp312-cp312-manylinux2014_x86_64.whl

Download URL xlite-0.2.0rc2-cp312-cp312-manylinux2014_x86_64.whl
Size 1.2 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
0c3efb08e85dd57b1ecaed4b9187feefedb938f2cbfb1639e2265e51cc140135
BLAKE2b-256 checksum
How to use checksums
c0ce11f1d57903d1e5ce4a45c3e80947c1f7d57570e7df6ada35a37092fddf8c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.11

Release files / xlite-0.2.0rc2-cp312-cp312-manylinux2014_aarch64.whl

Download URL xlite-0.2.0rc2-cp312-cp312-manylinux2014_aarch64.whl
Size 1.1 MB
Tags CPython 3.12 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
a0759345b1d11f0286a4a15d167ee6b89a05412e243f36644ca13ee6ef8b16a0
BLAKE2b-256 checksum
How to use checksums
ea844f7fa5e4cb590b8765b9b9c8eebf7d94431a9d183f8e1a04d7520df6652c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Release files / xlite-0.2.0rc2-cp311-cp311-manylinux2014_x86_64.whl

Download URL xlite-0.2.0rc2-cp311-cp311-manylinux2014_x86_64.whl
Size 1.2 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
a9c0e2c48119c35637b7d0b51a75ca1eb63ba993f87065f9a09a7ee71e2f4305
BLAKE2b-256 checksum
How to use checksums
515cdd6c819ba978ed062981bfe303d8bc133e32c9359159a4b04da2121c2475
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.11

Release files / xlite-0.2.0rc2-cp311-cp311-manylinux2014_aarch64.whl

Download URL xlite-0.2.0rc2-cp311-cp311-manylinux2014_aarch64.whl
Size 1.1 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
d43397dd5d44d86d81d01e732e1ce94c96fd7b4c832115daea868ed459d24598
BLAKE2b-256 checksum
How to use checksums
8edceddf881c206e5dc869b70774106a346ac714b18ea37e976b18eeaddf53c7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Security for xlite 0.2.0rc2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page