nm-vllm 0.6.3.0

A high-throughput and memory-efficient inference and serving engine for LLMs

nm-vllm

Overview

This repo nm-vllm-ent contains all the source for the Neuralmagic Enterprise Edition of vllm. The nm-vllm packages built from this repo are supported enterprise distributions of vLLM. Packages are versioned Python wheels and docker images. These are released as "production level" official releases and "beta level" Nightly's.

Official releases are made at the discretion of Neuralmagic, but typically track with vllm releases. These wheels are available via "public pypi" as well as "nm-pypi".

Nightly's are released every night given green runs in automation. The wheels are available at "nm-pypi".

Installation

PyPI

The nm-vllm PyPi package includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source.

Install it using pip:

pip install nm-vllm --extra-index-url https://pypi.neuralmagic.com/simple

To utilize the weight sparsity features, include the optional sparse dependencies.

pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple

You can also build and install nm-vllm from source (this will take ~10 minutes):

git clone https://github.com/neuralmagic/nm-vllm.git
cd nm-vllm
pip install -e .[sparse] --extra-index-url https://pypi.neuralmagic.com/simple

Docker

The nm-vllm container registry includes premade docker images.

Launch the OpenAI-compatible server with:

MODEL_ID=Qwen/Qwen2-0.5B-Instruct
docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-openai:latest --model $MODEL_ID

Models

Neural Magic maintains a variety of optimized models on our Hugging Face organization profiles:

• neuralmagic
• nm-testing