lk_moe
lk_moe is a highly optimized MOE hybrid inference engine that serves as the computational core for MOE layer execution. It is designed to maximize resource utilization across CPU, memory, and GPU, enabling large-scale MOE models to run efficiently beyond GPU memory constraints.
Key Capabilities
- Hybrid Compute Backend: Each MOE layer can dynamically choose between the original GPU computation path or lk_moe's hybrid inference path, offering flexible deployment strategies.
- CPU-GPU Collaborative Inference: Distributes expert computation across CPU (via NUMA-optimized threading) and GPU, reducing GPU memory requirements while maintaining high throughput.
- NUMA-Aware Parallelism: Optimized thread binding and memory allocation across NUMA nodes, achieving cross-node communication as low as 3% and L3 cache hit rates above 50%.
- Quantization Kernel Support: Native support for multiple quantization formats including BF16, FP8, NVFP4, MXFP4, and AWQ 4-bit, with efficient kernel implementations.
- Memory + VRAM Load Balancing: Total model footprint spans both VRAM and system memory, allowing models larger than GPU memory to run without performance collapse.
Framework Integration
lk_moe integrates with popular LLM serving frameworks as a drop-in MOE computation backend:
| Framework | Integration | Description |
|---|---|---|
| vLLM | LvLLM | lk_moe as MOE layer backend for vLLM serving |
| sglang | Lsglang | lk_moe as MOE layer backend for sglang serving |
| DeepSeek V4 | Lvllmds4 | Specialized integration for DeepSeek V4 (SM120) |
| DeepSeek V4 | Lvllmds4-x | Specialized integration for DeepSeek V4 (SM80+) |
Installation
bash pip install lk-moe
License
lk_moe is provided under a proprietary software license agreement. See LICENSE for terms and conditions. Third-party open-source components and their respective licenses are listed in THIRD_PARTY_LICENSES.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file lk_moe-2.3.3-cp312-cp312-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: lk_moe-2.3.3-cp312-cp312-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 21.7 MB
- Tags: CPython 3.12, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7b207fdca45a5866e9f93a98b042c7992b5e777507a8d1f29567a84272143585
|
|
| MD5 |
c648c948cfa1a62f5676bfae2159bdd9
|
|
| BLAKE2b-256 |
ddf61accb05e13aa2fb321c60898f301d269f6a779a59d3a41d9ab866534b2ae
|
