vllm-plugin-meralion2 0.1.2.post2.dev3

A vLLM plugin to register the MERaLiON-2-10B model architecture with vLLM’s plugin system.

MERaLiON2 vLLM Plugin

Licence

MERaLiON-Public-Licence-v2

Set up Environment

This vLLM plugin for MERaLiON2 requires transformers version 4.50.1. It supports vLLM version 0.6.5 ~ 0.7.3 (V0 engine), and 0.8.5 ~ 0.8.5.post1 (V1 engine).

pip install transformers==4.50.1
pip install vllm==0.6.5

Install the MERaLiON2 vLLM plugin.

python install vllm-plugin-meralion2

It's strongly recommended to install flash-attn for better memory and gpu utilization.

pip install flash-attn --no-build-isolation

Offline Inference

Refer to offline_example.py for offline inference example.

OpenAI-compatible Serving

Refer to openai_serve_example.sh for openAI-compatible serving example.

To call the server, you can refer to openai_client_example.py.

Alternatively, you can try calling the server with curl, refer to openai_client_curl.sh.