gk-server

Local OpenAI-compatible LLM server GUI for GGUF models, packaged for Python — the server half of the unified ggk package. The GUI runs in your browser against a local backend; inference is served by the gguf-server C/C++ engine, compiled during pip install and bundled with the package as a single binary. The engine evaluates its graphs on gk, an independent tensor library — there is no llama.cpp checkout and no ggml anywhere in the tree. Model and template files are referenced by filesystem path through a built-in file browser — nothing is uploaded or copied to temp storage.

Install

pip install gk-server

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on Windows: MSVC Build Tools). The engine source is resolved from the vendored vendor/server copy (see scripts/vendor_engine.py) or GK_SERVER_ENGINE_DIR. That tree is self-contained — it carries the gk kernels, the GGUF runtime, the common layer and the HTTP server — so the build compiles the server binary and nothing else.

The vendor/server/gk kernels are shared verbatim with the gk-diffuser package; both are vendored from the same unified ggk engine tree, so the two packages always compute with the same gk.

GPU and accelerator backends

The default build is CPU-only (on macOS, Metal is on by default — no flag needed). Backends are opt-in and can be requested with an environment variable or a CMake define — the env var is usually easier to pass through pip:

GK_SERVER_CUDA=1   pip install gk-server    # NVIDIA (needs the CUDA toolkit)
GK_SERVER_HIP=1    pip install gk-server    # AMD (needs ROCm/HIP)
GK_SERVER_VULKAN=1 pip install gk-server    # cross-vendor (needs the Vulkan SDK)

CMAKE_ARGS="-DGK_SERVER_CUDA=ON" pip install gk-server   # equivalent

Available: CUDA, HIP, VULKAN, METAL. Each option maps to the gk backend of the same name, and the finer-grained GK_* knobs (GK_NATIVE, GK_CUDA_ARCHITECTURES, …) can still be passed straight through as -DGK_<NAME>=….

A CUDA build works its own architecture list out from nvcc and the installed GPUs, and embeds PTX for the newest, so an unlisted card JITs rather than failing. A wheel built on one machine for another should still say what it targets, e.g. CMAKE_ARGS="-DGK_SERVER_CUDA=ON -DGK_CUDA_ARCHITECTURES=89".

HTTPS / OpenSSL

Off by default. OpenSSL is only needed to download models over HTTPS (-hf / URL arguments); this package always hands the engine local file paths. Turn it on with GK_SERVER_OPENSSL=1 if you script the engine directly and want HTTPS downloads.

Run

gk-server              # GUI on http://127.0.0.1:8642, opens the browser
python -m gk_server    # same thing
gk-server --port 0     # pick a free port; --no-browser to stay headless

The LLM server the GUI manages defaults to port 8888 and exposes the usual OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/messages, /props, /health, …).

The engine is directly scriptable from the CLI too:

gk-server engine -- --model model.gguf --port 8888
gk-server engine -- --help