Environment
To create a Python virtual environment, use the command:
conda env create -f environment.yml
Installation
pip install contextflow
Supported Models
The following LLM models are supported:
- Gemma Family
- Qwen Family
- YandexGPT Family
LLM backends
The following LLM backends are supported:
- Llama.cpp Server API
Run Llama.CPP Server backend
llama.cpp/build/bin/llama-server -m model_q5_k_m.gguf -ngl 99 -fa -np 2 -c 8192 --host 0.0.0.0 --port 8000
Install CUDA toolkit for Llama.cpp compilation
Please note that the toolkit version must match the driver version. The driver version can be found using the nvidia-smi command. Аor example, to install toolkit for CUDA 12.4 you need to run the following commands:
CUDA_TOOLKIT_VERSION=12-4
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt -y install cuda-toolkit-${CUDA_TOOLKIT_VERSION}
echo -e '
export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
' >> ~/.bashrc
Set GPU Max Temp
nvidia-smi -pm 1
sudo nvidia-smi -gtt 80
nvidia-smi -q | grep Target
1 maintainer
Metadata
1 maintainer
Release files for contextflow 1.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| contextflow-1.10.0.tar.gz | 7.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| contextflow-1.10.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.1 kB
Release files / contextflow-1.10.0.tar.gz
| Download URL | contextflow-1.10.0.tar.gz |
|---|---|
| Size | 7.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f6a21402989a11b5aed01481b61ab039070edf3661a39dc0808796469ad8f6ea
|
|
BLAKE2b-256 checksum How to use checksums |
26b92bb2f294516c70acc4248a4730e93238467d131d053d44b51cbe042de16c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / contextflow-1.10.0-py3-none-any.whl
| Download URL | contextflow-1.10.0-py3-none-any.whl |
|---|---|
| Size | 8.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
68793717cfc107fc16d1d2b8e1bfdc3174e61e934d3c2215a6050a62f9799902
|
|
BLAKE2b-256 checksum How to use checksums |
174734ac71a6127bb47ed793ba38d0f800567278ff95c60849cd03155efcba7c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
