local-llm 0.0.10

Local-LLM is a llama.cpp server in Docker with OpenAI Style Endpoints.

Local-LLM

• Dockerhub
• GitHub

Local-LLM is a llama.cpp server in Docker with OpenAI Style Endpoints that allows you to send the model name as the name of the model as it appears in the model list, for example Mistral-7B-OpenOrca. It will automatically download the model from Hugging Face if it isn't already downloaded and configure the server for you. It automatically configures the server based on your CPU, RAM, and GPU. It is designed to be as easy as possible to get started with running local models.

Table of Contents 📖

• Local-LLM
  • Table of Contents 📖
  • Run with Docker
    • Prerequisites
    • Run without NVIDIA GPU support
    • Run with NVIDIA GPU support
  • OpenAI Style Endpoint Usage
  • Shout Outs

Run with Docker

You can choose to run with Docker or Docker Compose. Both are not needed. Instructions to run with Docker Compose can be found here.

Replace the environment variables with your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.

• LOCAL_LLM_API_KEY - The API key to use for the server. If not set, the server will not require an API key.
• THREADS - The number of threads to use. Default is your CPU core count minus 1.
• BATCH_SIZE - The batch size to use for batch generation. Default is 512.

The following are only applicable to NVIDIA GPUs:

• GPU_LAYERS - The number of layers to use on the GPU. Default is 0.
• MAIN_GPU - The GPU to use for the main model. Default is 0.

Prerequisites

• Docker

Run without NVIDIA GPU support

docker pull joshxt/local-llm:cpu
docker run -d --name local-llm -p 8091:8091 joshxt/local-llm:cpu -e THREADS="10" -e BATCH_SIZE="512" -e LOCAL_LLM_API_KEY=""

Run with NVIDIA GPU support

If you're using an NVIDIA GPU, you can use the CUDA version of the server.

docker pull joshxt/local-llm:cuda
docker run -d --name local-llm -p 8091:8091 --gpus all joshxt/local-llm:cuda -e THREADS="10" -e BATCH_SIZE="512" -e GPU_LAYERS="0" -e MAIN_GPU="0" -e LOCAL_LLM_API_KEY=""

OpenAI Style Endpoint Usage

OpenAI Style endpoints available at http://localhost:8091/ by default. Documentation can be accessed at that url when the server is running. There are examples for each of the endpoints in the Examples Jupyter Notebook.

Shout Outs

• ggerganov/llama.cpp - For constantly improving the ability for anyone to run local models. It is one of my favorite and most exciting projects on GitHub.
• abetlen/llama-cpp-python - For making it easy to extend the functionality of llama.cpp in Python.
• TheBloke - For helping enable the ability to run local models by quantizing them and sharing them with a great readme on how to use them in every repository.
• Meta - For the absolutely earth shattering open source releases of the LLaMa models and all other contributions they have made to Open Source.
• OpenAI - For setting good standards for endpoints and making great models.
• Hugging Face - For making it easy to use and share models.
• As much as I hate to do it, I can't list all of the amazing people building and fine tuning local models, but you know who you are. Thank you for all of your hard work and contributions to the community!