Local-LLM
Local-LLM is a simple llama.cpp server that easily exposes a list of local language models to choose from to run on your own computer. It is designed to be as easy as possible to get started with running local models. It automatically handles downloading the model of your choice and configuring the server based on your CPU, RAM, and GPU. It also includes OpenAI Style endpoints for easy integration with other applications. Additional functionality is built in for voice cloning text to speech and a voice to text for easy voice communication entirely offline after the initial setup.
Prerequisites
- Git
- PowerShell 7.X
- Docker Desktop (Windows or Mac)
Additional Linux Prerequisites
Installation
git clone https://github.com/Josh-XT/Local-LLM
cd Local-LLM
Environment Setup
Expand Environment Setup if you would like to modify the default environment variables, otherwise skip to Usage. All environment variables are optional and have useful defaults. Change the default model that starts with Local-LLM in your .env file.
Environment Setup (Optional)
None of the values need modified in order to run the server. If you are using an NVIDIA GPU, I would recommend setting the GPU_LAYERS and MAIN_GPU environment variables. If you plan to expose the server to the internet, I would recommend setting the LOCAL_LLM_API_KEY environment variable for security. THREADS is set to your CPU thread count minus 2 by default, if this causes significant performance issues, consider setting the THREADS environment variable manually to a lower number.
Modify the .env file to your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
Replace the environment variables with your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.
LOCAL_LLM_API_KEY- The API key to use for the server. If not set, the server will not require an API key when accepting requests.DEFAULT_MODEL- The default model to use when no model is specified. Default isphi-2-dpo.WHISPER_MODEL- The model to use for speech-to-text. Default isbase.en.AUTO_UPDATE- Whether or not to automatically update Local-LLM. Default istrue.THREADS- The number of CPU threads Local-LLM is allowed to use. Default isyour CPU thread count minus 2.GPU_LAYERS(Only applicable to NVIDIA GPU) - The number of layers to use on the GPU. Default is0. Local-LLM will automatically determine the optimal number of layers to use based on your GPU's memory if it is set to 0 and you have an NVIDIA GPU.MAIN_GPU(Only applicable to NVIDIA GPU) - The GPU to use for the main model. Default is0.
Usage
./start.ps1
For examples on how to use the server to communicate with the models, see the Examples Jupyter Notebook.
OpenAI Style Endpoint Usage
OpenAI Style endpoints available at http://<YOUR LOCAL IP ADDRESS>:8091/v1/ by default. Documentation can be accessed at that http://localhost:8091 when the server is running.
1 maintainer
Metadata
1 maintainer
Release files for local-llm 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| local-llm-0.1.1.tar.gz | 17.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| local_llm-0.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 27.9 kB
Release files / local-llm-0.1.1.tar.gz
| Download URL | local-llm-0.1.1.tar.gz |
|---|---|
| Size | 17.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a1b80761ec74e1d2fc5120d6eb10350a9c74b8282c705aa729e43203b17bc4c4
|
|
BLAKE2b-256 checksum How to use checksums |
b3d040137a7636e81062f5a7f835e332351808002620e376a4b96de4858d584c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.7
|
Release files / local_llm-0.1.1-py3-none-any.whl
| Download URL | local_llm-0.1.1-py3-none-any.whl |
|---|---|
| Size | 10.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1c8b07526addc8a711dfd7ae8ff00f5ac9aaa745aaa9d2cfc1591192f1c41b8a
|
|
BLAKE2b-256 checksum How to use checksums |
2bcbaa50cf01cdbfe455c7c5d511001362d35014e33b6cf02120cc6637651f77
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.7
|
