Youtube Autonomous FastAPI Docker Llama.cpp Module

The module that is providing the functionality related to the Llama.cpp models hub (having the models and using them) through a FastAPI that is included and isolated in a Docker container.

This module is meant to be exposed as a container inside the internal network, to be connected with its own FastAPI that is exposing the functionality outside.

Endpoints

GET

No endpoints by now.

POST

No endpoints by now.

Instructions

I've followed these steps to make llama.cpp available in my laptop as a container running with cuda, and I've adapted this workflow to this project so its done automatically:

1. I will use a gemma4 image that is built for CUDA, so I will make sure that I have CUDA properly installed and detected by docker. Execute the following command and it will show your Nvidia GPU if installed and detected: $docker run --rm --gpus all nvidia/cuda:12.9.1-runtime-ubuntu24.04 nvidia-smi. If the output is positive, you can go on and use your GPU. Once you've confirmed that you can or cannot, delete the image that has been downloaded with the command. You don't need it.

2. Create a models folder in which we will store the models we will use (I will use an external SSD to save space). My folder is D:/llama/models.

3. Download the GGUF model that we need (check the specifications of your PC and choose the right one), in the cmd inside the models folder you created in the 2nd step: $huggingface-cli download unsloth/gemma-4-E2B-it-GGUF gemma-4-E2B-it-UD-Q4_K_XL.gguf --local-dir ./

4. Download the llamacpp version that is using CUDA, being in the cmd inside the models: $docker run --rm --gpus all -p 8080:8080 -v "${PWD}:/models" ghcr.io/ggml-org/llama.cpp:server-cuda -m --host 0.0.0.0 -ngl 999

Extra

If you download and execute llamacpp by itself, you'll a web client like with ChatGPT in which you can have a chat with your agent (it should probably be accesible through the http://localhost:8080 url). I'm not using it like this, so I wan't it accessible just by my custom endpoints through my APIs.