server to serve mlx model as an OpenAI compatible API
Project description
mlx-llm-server
This guide will help you set up the MLX-LLM server to serve the model as an OpenAI compatible API.
Quick Start
Installation
Before starting the MLX-LLM server, install the server package from PyPI:
pip install mlx-llm-server
Start the Server
mlx-llm-server --model-path <path-to-your-model>
Arguments
--model-path
: The path to the mlx model weights, tokenizer, and config. This argument is required.
Host and Port Configuration
The server will start on the host and port specified by the environment variables HOST
and PORT
. If these are not set, it defaults to 127.0.0.1:8080
.
To start the server on a different host or port, set the HOST
and PORT
environment variables before starting the server. For example:
export HOST=0.0.0.0
export PORT=5000
mlx-llm-server --model-path <path-to-your-model>
The MLX-LLM server can serve both Hugging Face format models and quantized MLX models. You can find these models at the MLX Community on Hugging Face.
Development Setup Guide
Miniconda Installation
For Apple Silicon users, install Miniconda natively with these commands:
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh
Conda Environment Setup
After Miniconda installation, create a dedicated conda environment for MLX-LLM:
conda create -n mlx-llm python=3.10
conda activate mlx-llm
Installing Dependencies
With the mlx-llm
environment activated, install the necessary dependencies using the following command:
pip install -r requirements.txt
Testing the API with curl
You can test the API using the curl
command. Here's an example:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-3.5-turbo",
"stop":["<|im_end|>"],
"messages": [
{
"role": "user",
"content": "Write a limerick about python exceptions"
}
]
}'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mlx_llm_server-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 949baaf29f3833b313009c9ca89a081a7d3dd73e3b362cfcaf6890cdcf05e556 |
|
MD5 | e41503b780f552556bec2c4b15c5f4e5 |
|
BLAKE2b-256 | 2496eb1c29c404db513b3fa4699edaec5c4a955a0d5d839093790bf9f02202e7 |