Skip to main content

vkserve

An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).

Install

On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the system Vulkan loader (libvulkan.so.1) and a GPU driver.

On macOS arm64, pip install vkserve is enough too. The wheel bundles MoltenVK (Vulkan on Metal) and the Vulkan loader, so it needs no Vulkan SDK. Set VK_DRIVER_FILES to use another driver.

Other systems build the engine from a source checkout:

git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine

make engine needs compute-ml 071d587 (0.4.73) or newer. It stops with a message that names the missing function when the checkout is older. Update it with git -C ../compute-ml pull.

Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.

Run it

  1. Build the C engine. It compiles compute-ml from ../compute-ml.

    make engine
    
  2. Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.

    uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000
    

    The model id in the API is the last part of the name (Qwen3-0.6B); --name changes it.

    Pick the GPU with --device (an index or part of the name). vkserve --list-devices shows them. Repeat --device to run the model on several GPUs: requests go to the least busy one, and a GPU that cannot load the model is skipped.

    uv run vkserve --list-devices
    uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000
    

    vkserve --list-models shows the models already in the Hugging Face cache, the ones vkserve can serve first.

    uv run vkserve --list-models
    

    Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a tokenizer.json. A bare .gguf file is not supported yet. Several models in one server: repeat --model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096].

  3. Send a request.

    curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
        -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
    

Use it from opencode

See examples/opencode: a config, a small agent, and the commands.

Configuration

One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.

uv run vkserve -t -c vkserve.toml                 # check the file, load nothing
uv run vkserve -c vkserve.toml                    # run it
uv run vkserve -c vkserve.toml --port 9000        # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2   # change one model of the file
uv run vkserve --print-config -c vkserve.toml     # show every effective setting
  • [server] keys are plain flags: --host, --port, --log-level, --keep-going, --max-body-bytes, --pid-file, --log-file, --drain-timeout.
  • --model NAME starts a model block. The flags after it apply to that block: --path, --dtype, --slots, --max-len, --prefill-chunk, --max-queue, --device (repeat for several devices), --temperature, --top-p, --top-k.
  • --set server.port=9000 and --set model.qwen.slots=2 reach any key.
  • An unknown key is an error with a hint: unknown key 'slot' ... (did you mean 'slots'?).

Run it as a service

The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.

Send signals with kill, or with -s (it reads pid_file):

Signal -s What it does
SIGHUP reload Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing.
SIGTERM, SIGQUIT, SIGINT stop, quit Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once.
SIGUSR1 reopen Reopen log_file (after logrotate moved it).

host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart. A reload that changes them logs a warning and ignores them.

uv run vkserve -s reload -c vkserve.toml

A systemd unit is in examples/vkserve.service.

Features

  • POST /v1/chat/completions and /v1/completions, streaming as server-sent events.
  • Tool calls: tools in, OpenAI tool_calls out, tool messages back in.
  • Several requests in one batched forward pass per step.
  • Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
  • Operations: 429 when the queue is full, X-Request-ID, Prometheus /metrics per device.
  • Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
  • Attention over long contexts (tested to 20,000 keys).

More: docs/serving.md (design and measurements).

Metadata

Release files for vkserve 0.1.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for vkserve 0.1.20
File Interpreter ABI Platform Actions
vkserve-0.1.20-py3-none-manylinux_2_34_x86_64.whl Python 3 none Linux glibc 2.34+ x86-64 Details for vkserve-0.1.20-py3-none-manylinux_2_34_x86_64.whl
vkserve-0.1.20-py3-none-macosx_15_0_arm64.whl Python 3 none macOS 15.0+ ARM64 Details for vkserve-0.1.20-py3-none-macosx_15_0_arm64.whl

Total release size: 1.6 MB

Release files / vkserve-0.1.20-py3-none-manylinux_2_34_x86_64.whl

Download URL vkserve-0.1.20-py3-none-manylinux_2_34_x86_64.whl
Size 715.1 kB
Tags Linux glibc 2.34+ x86-64 Python 3
SHA-256 checksum
How to use checksums
3f6f9c0594af3feee1ae6752d273904ed2c55d85ad6fcc9eaad44759a89a8f6d
BLAKE2b-256 checksum
How to use checksums
0ceca9fc5885e1be44bc6dce52c107c7152fefc3b1dd4202ad12cc50998bbd3b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / vkserve-0.1.20-py3-none-macosx_15_0_arm64.whl

Download URL vkserve-0.1.20-py3-none-macosx_15_0_arm64.whl
Size 891.9 kB
Tags Python 3 macOS 15.0+ ARM64
SHA-256 checksum
How to use checksums
4538df5e121b93791f0e883345f54643f05dee16362aaaf62f16f205b2cdb7a9
BLAKE2b-256 checksum
How to use checksums
b7382aa1f8f0e7209f01ca6eddd3f19055493fb2ac63907c97c479b9e0c2783f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for vkserve 0.1.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page