A hybrid chatbot - LLM side.
Project description
WAFL-llm
WAFL is built to run as a two-part system. Both can be installed on the same machine. This is the LLM side of the WAFL project.
LLM side (needs a GPU)
This is a model server for the speech-to-text model, the LLM, the embedding system, and the text-to-speech model.
Installation
In order to quickly run the LLM side, you can use the following installation commands:
pip install wafl-llm
wafl-llm start
which will use the default models and start the server on port 8080.
The installation will require MPI and Java installed on the system. One can install both with the following commands
sudo apt install libopenmpi-dev
sudo apt install default-jdk
Configuration
A use-case specific configuration can be set by creating a config.json
file in the path where wafl-llm start
is executed.
The file should look like this (the default configuration)
{
"llm_model": "mistralai/Mistral-7B-Instruct-v0.1",
"speaker_model": "facebook/fastspeech2-en-ljspeech",
"whisper_model": "fractalego/personal-whisper-distilled-model",
"sentence_embedder_models": "TaylorAI/gte-tiny"
}
The models are downloaded from the HugggingFace repository. Any other compatible model should work.
Docker
A docker image can be used to run it as in the following:
$ docker run -p8080:8080 --env NVIDIA_DISABLE_REQUIRE=1 --gpus all fractalego/wafl-llm:latest
or one can clone this repository and run the following
docker/build.sh
docker run -p8080:8080 --env NVIDIA_DISABLE_REQUIRE=1 --gpus all wafl-llm
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for wafl_llm-0.0.83-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 715d4218bf3206049be54557662882f236f705c54195d007bae0d62236e5b3e2 |
|
MD5 | 999bf80d0fda3b195bd95c31fafa181c |
|
BLAKE2b-256 | 77b375d82b07a95d734b68c25adb421250c2fe6c13339348fbd4d950bd0341c5 |