lk-moe 2.3.1

A highly optimized MOE hybrid inference engine for CPU-GPU collaborative execution of large-scale MOE models.

lk_moe

lk_moe is a highly optimized MOE hybrid inference engine that serves as the computational core for MOE layer execution. It is designed to maximize resource utilization across CPU, memory, and GPU, enabling large-scale MOE models to run efficiently beyond GPU memory constraints.

Key Capabilities

• Hybrid Compute Backend: Each MOE layer can dynamically choose between the original GPU computation path or lk_moe's hybrid inference path, offering flexible deployment strategies.
• CPU-GPU Collaborative Inference: Distributes expert computation across CPU (via NUMA-optimized threading) and GPU, reducing GPU memory requirements while maintaining high throughput.
• NUMA-Aware Parallelism: Optimized thread binding and memory allocation across NUMA nodes, achieving cross-node communication as low as 3% and L3 cache hit rates above 50%.
• Quantization Kernel Support: Native support for multiple quantization formats including BF16, FP8, NVFP4, MXFP4, and AWQ 4-bit, with efficient kernel implementations.
• Memory + VRAM Load Balancing: Total model footprint spans both VRAM and system memory, allowing models larger than GPU memory to run without performance collapse.

Framework Integration

lk_moe integrates with popular LLM serving frameworks as a drop-in MOE computation backend:

Framework	Integration	Description
vLLM	LvLLM	lk_moe as MOE layer backend for vLLM serving
sglang	Lsglang	lk_moe as MOE layer backend for sglang serving
DeepSeek V4	Lvllmds4	Specialized integration for DeepSeek V4 (SM120)

Installation

bash pip install lk-moe

License

lk_moe is provided under a proprietary software license agreement. See LICENSE for terms and conditions. Third-party open-source components and their respective licenses are listed in THIRD_PARTY_LICENSES.