Local-LLM is a llama.cpp server in Docker with OpenAI Style Endpoints.
Project description
Local-LLM
Local-LLM is a llama.cpp server in Docker with OpenAI Style Endpoints that allows you to send the model name as the name of the model as it appears in the model list, for example Mistral-7B-OpenOrca
. It will automatically download the model from Hugging Face if it isn't already downloaded and configure the server for you. It automatically configures the server based on your CPU, RAM, and GPU. It is designed to be as easy as possible to get started with running local models.
Table of Contents 📖
Run with Docker
You can choose to run with Docker or Docker Compose. Both are not needed. Instructions to run with Docker Compose can be found here.
Replace the environment variables with your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
LOCAL_LLM_API_KEY
- The API key to use for the server. If not set, the server will not require an API key.THREADS
- The number of threads to use. Default isyour CPU core count minus 1
.
The following are only applicable to NVIDIA GPUs:
GPU_LAYERS
- The number of layers to use on the GPU. Default is0
.MAIN_GPU
- The GPU to use for the main model. Default is0
.
Prerequisites
Run without NVIDIA GPU support
Modify the THREADS
environment variable to your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
docker pull joshxt/local-llm:cpu
docker run -d --name local-llm -p 8091:8091 joshxt/local-llm:cpu -e THREADS="10" -e LOCAL_LLM_API_KEY="" -v ./models:/app/models
Run with NVIDIA GPU support
If you're using an NVIDIA GPU, you can use the CUDA version of the server. You must have the NVIDIA Container Toolkit installed if using NVIDIA GPU.
Modify the GPU_LAYERS
, MAIN_GPU
, and THREADS
environment variables to your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
docker pull joshxt/local-llm:cuda
docker run -d --name local-llm -p 8091:8091 --gpus all joshxt/local-llm:cuda -e THREADS="10" -e GPU_LAYERS="20" -e MAIN_GPU="0" -e LOCAL_LLM_API_KEY="" -v ./models:/app/models
Run with Docker Compose
You can choose to run with Docker Compose or Docker. Both are not needed.
Update the .env
file with your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
Run without NVIDIA GPU support with Docker Compose
docker-compose pull
docker-compose up
Run with NVIDIA GPU support with Docker Compose
docker-compose -f docker-compose-cuda.yml pull
docker-compose -f docker-compose-cuda.yml up
OpenAI Style Endpoint Usage
OpenAI Style endpoints available at http://<YOUR LOCAL IP ADDRESS>:8091/v1
by default. Documentation can be accessed at that http://localhost:8091 when the server is running. There are examples for each of the endpoints in the Examples Jupyter Notebook.
Shout Outs
- ggerganov/llama.cpp - For constantly improving the ability for anyone to run local models. It is one of my favorite and most exciting projects on GitHub.
- abetlen/llama-cpp-python - For making it easy to extend the functionality of llama.cpp in Python.
- TheBloke - For helping enable the ability to run local models by quantizing them and sharing them with a great readme on how to use them in every repository.
- Meta - For the absolutely earth shattering open source releases of the LLaMa models and all other contributions they have made to Open Source.
- OpenAI - For setting good standards for endpoints and making great models.
- Hugging Face - For making it easy to use and share models.
- As much as I hate to do it, I can't list all of the amazing people building and fine tuning local models, but you know who you are. Thank you for all of your hard work and contributions to the community!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for local_llm-0.0.23-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 254083e498cbefd400564de7b0eab31702ef116568320d2992114fcbe418edc1 |
|
MD5 | 827446f44e22d8cbef11d7c3f9c125d2 |
|
BLAKE2b-256 | 9eb25f7ffb6f64da8b0fb898eb5b2d9d158198c5657963df6df09af76dcbcf67 |