gguf-server

Local LLM server GUI for GGUF models, packaged for Python. The GUI runs in your browser against a local backend; inference is served by the standalone gguf-server C/C++ engine, compiled during pip install and bundled with the package as a single binary. Model and template files are referenced by filesystem path through a built-in file browser — nothing is uploaded or copied to temp storage.

Install

pip install gguf-server

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on Windows: MSVC Build Tools). The engine source is resolved from the vendored vendor/server copy (see scripts/vendor_engine.py), a sibling ../server checkout, or GGUF_SERVER_ENGINE_DIR. That tree is self-contained — it carries its own ggml kernels, GGUF runtime, common layer and HTTP server, with no llama.cpp checkout and no external ggml — so the build compiles the server binary and nothing else.

GPU and accelerator backends

The default build is CPU-only. Backends are opt-in and can be requested with an environment variable or a CMake define — the env var is usually easier to pass through pip:

GGUF_SERVER_CUDA=1   pip install gguf-server    # NVIDIA (needs the CUDA toolkit)
GGUF_SERVER_HIP=1    pip install gguf-server    # AMD (needs ROCm/HIP)
GGUF_SERVER_VULKAN=1 pip install gguf-server    # cross-vendor (needs the Vulkan SDK)

CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON" pip install gguf-server   # equivalent

Available: CUDA, HIP, VULKAN, METAL. These are the kernels the engine tree actually ships (vendor/server/kernels/src); the backends llama.cpp has and this engine does not — SYCL, CANN, OpenCL, WebGPU, RPC, zDNN, ZenDNN, OpenVINO, Hexagon, MUSA, BLAS — are simply absent. Each option maps to the ggml option of the same name, and the finer-grained GGML_* knobs (GGML_NATIVE, GGML_AVX512, GGML_CUDA_FA_ALL_QUANTS, …) can still be passed straight through as -DGGML_<NAME>=…. On macOS, Metal is on by default (with the shader library embedded in the binary) — no flag needed.

CUDA builds compile kernels for many GPU architectures and can take a long time; restrict them to your own card to speed this up, e.g. CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89". The GPU runtime libraries (cuBLAS, ROCm, …) are linked dynamically and come from the toolkit you built against, so keep it installed.

HTTPS / OpenSSL

Off by default here (the engine's own default is on). OpenSSL is only needed to download models over HTTPS (-hf / URL arguments); this package always hands the engine local file paths. Leaving it off also avoids a build break on Windows: find_package(OpenSSL) can pick up a foreign-ABI install — an MSYS2/MinGW libcrypto alongside an MSVC toolchain puts MinGW headers on cl.exe's include path and fails httplib.cpp, download.cpp and hf-cache.cpp with winnt.h: fatal error C1189: No supported target architecture. Re-enable with GGUF_SERVER_OPENSSL=1 if you want in-server HTTPS downloads.

Embedded web UI

The engine embeds whatever static assets sit in GGUF_SERVER_UI_DIR (default vendor/server/ui/dist, which ships empty), so out of the box it is API-only and its own / returns 404. That has no effect here — this package serves its own GUI. Point GGUF_SERVER_UI_DIR at a bundle if you want the engine's UI as well.

Windows notes

Both toolchains work; which one you get depends on your shell, so check the CMAKE_CXX_COMPILER line in the configure output if a build misbehaves.

• MSVC — build from a Developer Command Prompt (vcvars64.bat) so cl.exe is on PATH. This is the primary Windows toolchain.
• MinGW / MSYS2 — used automatically when C:\msys64\mingw64\bin is on PATH and MSVC is not. Supported here: the build defines _WIN32_WINNT=0x0A00 (MinGW otherwise pins it to a Windows 7-era value, which makes the vendored cpp-httplib hard-error with "doesn't support Windows 8 or lower" and leaves ::CreateFile2 undeclared), and links the MinGW runtime statically so the binary doesn't need libstdc++-6.dll, libgcc_s_seh-1.dll or libwinpthread-1.dll beside it. Turn the latter off with GGUF_SERVER_MINGW_STATIC=0.

Two environment gotchas worth knowing:

• 'ninja' ... inappropriate file type or format — a pyenv-win shim is shadowing the real ninja. Point CMake at a real one, e.g. -DCMAKE_MAKE_PROGRAM="C:/Program Files (x86)/Microsoft Visual Studio/<ver>/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe".
• A failed configure leaves a cache behind in build/, and later builds reuse it. Delete the directory (rmdir /s /q build) before retrying after you change toolchain or options.

Usage

python -m gguf_server              # launch the GUI in the browser
gguf-server                        # same
gguf-server --port 8642 --no-browser

The GUI listens on port 8642; the LLM server it launches defaults to 8888.

GUI features (as in the desktop app's LLM panel):

• Server — start/stop, live status, copyable API base URL, active configuration summary, OpenAI-compatible endpoint reference, and a copyable/editable engine command line
• Model — .gguf model file, optional --mmproj vision projector, and chat template selection (auto-detect from metadata, file, or raw Jinja2)
• Settings — host/port, API key, model alias, context length, GPU layers, main GPU, per-GPU tensor split sliders, CPU threads, parallel slots, batch and micro-batch sizes, KV cache types, flash attention, continuous batching, mlock, mmap and verbose logging, plus hardware-derived "Recommended" defaults
• Presets — save the whole configuration, with JSON export/import
• Hardware — CPU, RAM/VRAM meters and per-GPU details, with live polling
• Logs — tail of the running server's log with auto-scroll

Engine CLI passthrough (runs the bundled gguf-server engine binary):

python -m gguf_server engine -- --model model.gguf --port 8888
gguf-server engine -- --help

How it works

• pip install compiles the engine (static ggml + llama + llama-common + mtmd linked into one executable) via scikit-build-core and installs it into the package's bin/ directory.
• gguf-server starts a stdlib HTTP server (default port 8642) serving the static GUI and a small JSON API, and opens the browser.
• Starting the server spawns one engine child process with its output redirected to a log file; a watcher thread polls the TCP port until it accepts connections, then reports the endpoint to the GUI. Stopping the GUI stops the child.
• File selection uses a server-side directory listing (/api/browse) so the GUI gets real filesystem paths — no drag & drop uploads of multi-GB models.