Skip to main content

vkserve

An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).

Install

On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the system Vulkan loader (libvulkan.so.1) and a GPU driver.

On macOS arm64, pip install vkserve is enough too. The wheel bundles MoltenVK (Vulkan on Metal) and the Vulkan loader, so it needs no Vulkan SDK. Set VK_DRIVER_FILES to use another driver.

Other systems build the engine from a source checkout:

git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine

make engine needs compute-ml 071d587 (0.4.73) or newer. It stops with a message that names the missing function when the checkout is older. Update it with git -C ../compute-ml pull.

Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.

Run it

  1. Build the C engine. It compiles compute-ml from ../compute-ml.

    make engine
    
  2. Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.

    uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000
    

    The model id in the API is the last part of the name (Qwen3-0.6B); --name changes it.

    Pick the GPU with --device (an index or part of the name). vkserve --list-devices shows them. Repeat --device to run the model on several GPUs: requests go to the least busy one, and a GPU that cannot load the model is skipped.

    uv run vkserve --list-devices
    uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000
    

    Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a tokenizer.json. A bare .gguf file is not supported yet. Several models in one server: repeat --model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096].

  3. Send a request.

    curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
        -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
    

Use it from opencode

See examples/opencode: a config, a small agent, and the commands.

Configuration

One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.

uv run vkserve -t -c vkserve.toml                 # check the file, load nothing
uv run vkserve -c vkserve.toml                    # run it
uv run vkserve -c vkserve.toml --port 9000        # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2   # change one model of the file
uv run vkserve --print-config -c vkserve.toml     # show every effective setting
  • [server] keys are plain flags: --host, --port, --log-level, --keep-going, --max-body-bytes, --pid-file, --log-file, --drain-timeout.
  • --model NAME starts a model block. The flags after it apply to that block: --path, --dtype, --slots, --max-len, --prefill-chunk, --max-queue, --device (repeat for several devices), --temperature, --top-p, --top-k.
  • --set server.port=9000 and --set model.qwen.slots=2 reach any key.
  • An unknown key is an error with a hint: unknown key 'slot' ... (did you mean 'slots'?).

Run it as a service

The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.

Send signals with kill, or with -s (it reads pid_file):

Signal -s What it does
SIGHUP reload Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing.
SIGTERM, SIGQUIT, SIGINT stop, quit Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once.
SIGUSR1 reopen Reopen log_file (after logrotate moved it).

host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart. A reload that changes them logs a warning and ignores them.

uv run vkserve -s reload -c vkserve.toml

A systemd unit is in examples/vkserve.service.

Features

  • POST /v1/chat/completions and /v1/completions, streaming as server-sent events.
  • Tool calls: tools in, OpenAI tool_calls out, tool messages back in.
  • Several requests in one batched forward pass per step.
  • Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
  • Operations: 429 when the queue is full, X-Request-ID, Prometheus /metrics per device.
  • Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
  • Attention over long contexts (tested to 20,000 keys).

More: docs/serving.md (design and measurements).

Metadata

Release files for vkserve 0.1.18

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for vkserve 0.1.18
File Interpreter ABI Platform Actions
vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl Python 3 none Linux glibc 2.34+ x86-64 Details for vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl
vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl Python 3 none macOS 15.0+ ARM64 Details for vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl

Total release size: 1.6 MB

Release files / vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl

Download URL vkserve-0.1.18-py3-none-manylinux_2_34_x86_64.whl
Size 713.1 kB
Tags Linux glibc 2.34+ x86-64 Python 3
SHA-256 checksum
How to use checksums
48d0773dcde8416137ab9f324c75e00e6b78c691ec5847aa71d37475819393fc
BLAKE2b-256 checksum
How to use checksums
b8422dda950e6fe3218156d913e377b36442e5b198d3708ce4c27dc59ab981ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl

Download URL vkserve-0.1.18-py3-none-macosx_15_0_arm64.whl
Size 889.9 kB
Tags Python 3 macOS 15.0+ ARM64
SHA-256 checksum
How to use checksums
a43ff4c1cc938b83287fd99986a75b051b08c2f420b108c5d6c7d28fc9d5e573
BLAKE2b-256 checksum
How to use checksums
79c755fda52e6936ded3e7d8266cbd28aab94143cc317d0601be751301d24061
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for vkserve 0.1.18

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page