gguf-server

Local LLM server GUI for GGUF models, packaged for Python. The GUI runs in your browser against a local backend; inference is served by llama.cpp's llama-server, compiled during pip install and bundled with the package as a single binary. Model and template files are referenced by filesystem path through a built-in file browser — nothing is uploaded or copied to temp storage.

Install

pip install gguf-server

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on Windows: MSVC Build Tools). The engine source is resolved from the vendored vendor/llama.cpp copy (see scripts/vendor_engine.py), a sibling ../llama.cpp checkout, or GGUF_SERVER_ENGINE_DIR. Only llama-server and the libraries it links against are compiled — the rest of llama.cpp's tools, tests and examples are skipped.

GPU and accelerator backends

The default build is CPU-only. Backends are opt-in and can be requested with an environment variable or a CMake define — the env var is usually easier to pass through pip:

GGUF_SERVER_CUDA=1   pip install gguf-server    # NVIDIA (needs the CUDA toolkit)
GGUF_SERVER_HIP=1    pip install gguf-server    # AMD (needs ROCm/HIP)
GGUF_SERVER_VULKAN=1 pip install gguf-server    # cross-vendor (needs the Vulkan SDK)
GGUF_SERVER_SYCL=1   pip install gguf-server    # Intel oneAPI

CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON" pip install gguf-server   # equivalent

Available: CUDA, HIP, METAL, VULKAN, SYCL, OPENCL, MUSA, WEBGPU, ZDNN, OPENVINO, HEXAGON, BLAS, RPC. Each maps to the ggml option of the same name; anything not listed can still be passed straight through as -DGGML_<NAME>=ON. On macOS, Metal is on by default (with the shader library embedded in the binary) — no flag needed.

CUDA builds compile kernels for many GPU architectures and can take a long time; restrict them to your own card to speed this up, e.g. CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89". The GPU runtime libraries (cuBLAS, ROCm, …) are linked dynamically and come from the toolkit you built against, so keep it installed.

HTTPS / OpenSSL

Off by default. llama-server only needs OpenSSL to download models over HTTPS (-hf / URL arguments); this package always hands it local file paths. Leaving it off also avoids a build break on Windows: llama.cpp's find_package(OpenSSL) can pick up a foreign-ABI install — an MSYS2/MinGW libcrypto alongside an MSVC toolchain puts MinGW headers on cl.exe's include path and fails httplib.cpp, download.cpp and hf-cache.cpp with winnt.h: fatal error C1189: No supported target architecture. Re-enable with GGUF_SERVER_OPENSSL=1 if you want in-server HTTPS downloads.

Embedded web UI

llama-server's own web UI (served at the model endpoint) is fetched prebuilt when the network allows. Pass -DGGUF_SERVER_BUILD_UI=ON to build it from source with npm instead, or ignore it entirely — this package ships its own GUI and the LLM server works either way.

Windows notes

Both toolchains work; which one you get depends on your shell, so check the CMAKE_CXX_COMPILER line in the configure output if a build misbehaves.

• MSVC — build from a Developer Command Prompt (vcvars64.bat) so cl.exe is on PATH. This is llama.cpp's primary Windows toolchain.
• MinGW / MSYS2 — used automatically when C:\msys64\mingw64\bin is on PATH and MSVC is not. Supported here: the build defines _WIN32_WINNT=0x0A00 (MinGW otherwise pins it to a Windows 7-era value, which makes the vendored cpp-httplib hard-error with "doesn't support Windows 8 or lower" and leaves ::CreateFile2 undeclared), and links the MinGW runtime statically so the binary doesn't need libstdc++-6.dll, libgcc_s_seh-1.dll or libwinpthread-1.dll beside it. Turn the latter off with GGUF_SERVER_MINGW_STATIC=0.

Two environment gotchas worth knowing:

• 'ninja' ... inappropriate file type or format — a pyenv-win shim is shadowing the real ninja. Point CMake at a real one, e.g. -DCMAKE_MAKE_PROGRAM="C:/Program Files (x86)/Microsoft Visual Studio/<ver>/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe".
• A failed configure leaves a cache behind in build/, and later builds reuse it. Delete the directory (rmdir /s /q build) before retrying after you change toolchain or options.

Usage

python -m gguf_server              # launch the GUI in the browser
gguf-server                        # same
gguf-server --port 8642 --no-browser

The GUI listens on port 8642; the LLM server it launches defaults to 8888.

GUI features (as in the desktop app's LLM panel):

• Server — start/stop, live status, copyable API base URL, active configuration summary, OpenAI-compatible endpoint reference, and a copyable/editable llama-server command line
• Model — .gguf model file, optional --mmproj vision projector, and chat template selection (auto-detect from metadata, file, or raw Jinja2)
• Settings — host/port, API key, model alias, context length, GPU layers, main GPU, per-GPU tensor split sliders, CPU threads, parallel slots, batch and micro-batch sizes, KV cache types, flash attention, continuous batching, mlock, mmap and verbose logging, plus hardware-derived "Recommended" defaults
• Presets — save the whole configuration, with JSON export/import
• Hardware — CPU, RAM/VRAM meters and per-GPU details, with live polling
• Logs — tail of the running server's log with auto-scroll

Engine CLI passthrough (runs the bundled llama-server binary):

python -m gguf_server engine -- --model model.gguf --port 8888
gguf-server engine -- --help

How it works

• pip install compiles llama-server (static ggml + llama + common linked into one executable) via scikit-build-core and installs it into the package's bin/ directory.
• gguf-server starts a stdlib HTTP server (default port 8642) serving the static GUI and a small JSON API, and opens the browser.
• Starting the server spawns one llama-server child process with its output redirected to a log file; a watcher thread polls the TCP port until it accepts connections, then reports the endpoint to the GUI. Stopping the GUI stops the child.
• File selection uses a server-side directory listing (/api/browse) so the GUI gets real filesystem paths — no drag & drop uploads of multi-GB models.