vkserve
An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).
Install
On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the
system Vulkan loader (libvulkan.so.1) and a GPU driver.
On macOS arm64, pip install vkserve is enough too. The wheel bundles MoltenVK (Vulkan on Metal)
and the Vulkan loader, so it needs no Vulkan SDK. Set VK_DRIVER_FILES to use another driver.
Other systems build the engine from a source checkout:
git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine
make engine needs compute-ml 071d587 (0.4.73) or newer. It stops with a message that names
the missing function when the checkout is older. Update it with git -C ../compute-ml pull.
Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.
Run it
-
Build the C engine. It compiles compute-ml from
../compute-ml.make engine -
Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.
uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000
The model id in the API is the last part of the name (
Qwen3-0.6B);--namechanges it.Pick the GPU with
--device(an index or part of the name).vkserve --list-devicesshows them. Repeat--deviceto run the model on several GPUs: requests go to the least busy one, and a GPU that cannot load the model is skipped.uv run vkserve --list-devices uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000
Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a
tokenizer.json. A bare.gguffile is not supported yet. Several models in one server: repeat--model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096]. -
Send a request.
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \ -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
Use it from opencode
See examples/opencode: a config, a small agent, and the commands.
Configuration
One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.
uv run vkserve -t -c vkserve.toml # check the file, load nothing
uv run vkserve -c vkserve.toml # run it
uv run vkserve -c vkserve.toml --port 9000 # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2 # change one model of the file
uv run vkserve --print-config -c vkserve.toml # show every effective setting
[server]keys are plain flags:--host,--port,--log-level,--keep-going,--max-body-bytes,--pid-file,--log-file,--drain-timeout.--model NAMEstarts a model block. The flags after it apply to that block:--path,--dtype,--slots,--max-len,--prefill-chunk,--max-queue,--device(repeat for several devices),--temperature,--top-p,--top-k.--set server.port=9000and--set model.qwen.slots=2reach any key.- An unknown key is an error with a hint:
unknown key 'slot' ... (did you mean 'slots'?).
Run it as a service
The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.
Send signals with kill, or with -s (it reads pid_file):
| Signal | -s |
What it does |
|---|---|---|
SIGHUP |
reload |
Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing. |
SIGTERM, SIGQUIT, SIGINT |
stop, quit |
Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once. |
SIGUSR1 |
reopen |
Reopen log_file (after logrotate moved it). |
host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart.
A reload that changes them logs a warning and ignores them.
uv run vkserve -s reload -c vkserve.toml
A systemd unit is in examples/vkserve.service.
Features
POST /v1/chat/completionsand/v1/completions, streaming as server-sent events.- Tool calls:
toolsin, OpenAItool_callsout,toolmessages back in. - Several requests in one batched forward pass per step.
- Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
- Operations:
429when the queue is full,X-Request-ID, Prometheus/metricsper device. - Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
- Attention over long contexts (tested to 20,000 keys).
More: docs/serving.md (design and measurements).
Metadata
Release files for vkserve 0.1.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl | Python 3 | none | Linux glibc 2.34+ x86-64 | Details for vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl |
| vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl | Python 3 | none | macOS 15.0+ ARM64 | Details for vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl |
Total release size: 1.6 MB
Release files / vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl
| Download URL | vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl |
|---|---|
| Size | 713.1 kB |
| Tags | Linux glibc 2.34+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
48d0773dcde8416137ab9f324c75e00e6b78c691ec5847aa71d37475819393fc
|
|
BLAKE2b-256 checksum How to use checksums |
b8422dda950e6fe3218156d913e377b36442e5b198d3708ce4c27dc59ab981ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl
| Download URL | vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl |
|---|---|
| Size | 889.9 kB |
| Tags | Python 3 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a43ff4c1cc938b83287fd99986a75b051b08c2f420b108c5d6c7d28fc9d5e573
|
|
BLAKE2b-256 checksum How to use checksums |
79c755fda52e6936ded3e7d8266cbd28aab94143cc317d0601be751301d24061
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for vkserve 0.1.18
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.