happy_vllm is a REST API for vLLM, production ready
Project description
📚 Documentation : https://france-travail.github.io/happy_vllm/
happy_vLLM is a REST API for vLLM which was developed with production in mind. It adds some functionalities to vLLM.
Installation
You can install happy_vLLM using pip:
pip install happy_vllm
Or build it from source:
git clone https://github.com/France-Travail/happy_vllm.git
cd happy_vllm
pip install -e .
Quickstart
Just use the entrypoint happy-vllm
(see arguments for a list of all possible arguments)
happy-vllm --model path_to_model --host 127.0.0.1 --port 5000 --model-name my_model
It will launch the API and you can directly query it for example with
curl 127.0.0.1:5000/v1/info
To get various information on the application or
curl 127.0.0.1:5000/v1/completions -d '{"prompt": "Hey,", "model": "my_model"}'
if you want to generate your first LLM response using happy_vLLM. See endpoints for more details on all the endpoints provided by happy_vLLM.
Deploy with Docker image
A docker image is available from the Github Container Registry :
docker pull ghcr.io/france-travail/happy_vllm:latest
See deploying_with_docker for more details on how to serve happy_vLLM with docker.
Swagger
You can reach the swagger UI at the /docs
endpoint (so for example by default at 127.0.0.1:5000/docs
). You will be displayed all the endpoints and examples on how to use them.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for happy_vllm-1.1.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9169f0bb2041466d6de578d71792bdb6e76f4794b8089db8210de91a76487935 |
|
MD5 | e9fd71d0cd13002dfdd7ac3043fb1dfc |
|
BLAKE2b-256 | f879c25b4b02c7bff8d9e67a6b8f3c8ef93d3d11e1b0a03993e0be6f9a494219 |