galadriel-node 0.0.1

No project description provided

Galadriel inference node

Installation

pip install -e .

Running with LLM

Setup ollama:

# Install ollama on unix natively:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3

# Or docker:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull llama3

Run the node:

galadriel node run

Production VM deployment

Make sure GPU exists etc

nvidia-smi

Run vLLM

python3 -m venv venv
source venv/bin/activate
pip install vllm

mkdir logs

HUGGING_FACE_HUB_TOKEN=<HUGGING_FACE_TOKEN> \
nohup vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
    --revision db1f81ad4b8c7e39777509fac66c652eb0a52f91 \
    --max-model-len 16384 \
    --gpu-memory-utilization 1 \
    --host localhost \
    --disable-frontend-multiprocessing \
    --port 11434 > logs/logs0.log 2>&1 &

Setup node

ssh-keygen -t rsa -b 4096
# Add public key to repo "deploy keys"
# clone repo
cd galadriel-node

# deactivate other venv 
# deactivate
python3 -m venv venv
source venv/bin/activate

pip install -e .

Run node

GALADRIEL_API_KEY=<API KEY> \
    GALADRIEL_RPC_URL=ws://34.78.190.171/v1/node \
    nohup galadriel node run > logs.log 2>&1 &