Skip to main content

vkserve

An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml. Status: prototype. The Linux x86_64 wheel on PyPI includes the C engine; other systems build it from source (see Install).

Install

On Linux x86_64, pip install vkserve is enough. The wheel includes the C engine. It needs the system Vulkan loader (libvulkan.so.1) and a GPU driver. Other systems, such as macOS, build the engine from a source checkout:

git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine

make engine needs compute-ml 37bb499 (0.4.54) or newer. It stops with a message that names the missing function when the checkout is older. Update it with git -C ../compute-ml pull.

Or point VKSERVE_ENGINE_LIB at an engine library you built elsewhere.

Run it

  1. Build the C engine. It compiles compute-ml from ../compute-ml.

    make engine
    
  2. Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.

    uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000
    

    The model id in the API is the last part of the name (Qwen3-0.6B); --name changes it.

    Pick the GPU with --device (an index or part of the name). vkserve --list-devices shows them. Repeat --device to run the model on several GPUs: requests go to the least busy one, and a GPU that cannot load the model is skipped.

    uv run vkserve --list-devices
    uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000
    

    Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a tokenizer.json. A bare .gguf file is not supported yet. Several models in one server: repeat --model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096].

  3. Send a request.

    curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
        -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
    

Use it from opencode

See examples/opencode: a config, a small agent, and the commands.

Configuration

One TOML file, and every key is also a flag. Flags win over the file. See examples/vkserve.toml.

uv run vkserve -t -c vkserve.toml                 # check the file, load nothing
uv run vkserve -c vkserve.toml                    # run it
uv run vkserve -c vkserve.toml --port 9000        # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2   # change one model of the file
uv run vkserve --print-config -c vkserve.toml     # show every effective setting
  • [server] keys are plain flags: --host, --port, --log-level, --keep-going, --max-body-bytes, --pid-file, --log-file, --drain-timeout.
  • --model NAME starts a model block. The flags after it apply to that block: --path, --dtype, --slots, --max-len, --prefill-chunk, --max-queue, --device (repeat for several devices), --temperature, --top-p, --top-k.
  • --set server.port=9000 and --set model.qwen.slots=2 reach any key.
  • An unknown key is an error with a hint: unknown key 'slot' ... (did you mean 'slots'?).

Run it as a service

The master process serves HTTP. Each model on each device runs in its own worker process. A worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds. The master keeps running.

Send signals with kill, or with -s (it reads pid_file):

Signal -s What it does
SIGHUP reload Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing.
SIGTERM, SIGQUIT, SIGINT stop, quit Stop accepting, wait up to drain_timeout for requests in progress, then exit. A second signal stops at once.
SIGUSR1 reopen Reopen log_file (after logrotate moved it).

host, port, max_body_bytes, pid_file, drain_timeout and keep_going need a restart. A reload that changes them logs a warning and ignores them.

uv run vkserve -s reload -c vkserve.toml

A systemd unit is in examples/vkserve.service.

Features

  • POST /v1/chat/completions and /v1/completions, streaming as server-sent events.
  • Tool calls: tools in, OpenAI tool_calls out, tool messages back in.
  • Several requests in one batched forward pass per step.
  • Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
  • Operations: 429 when the queue is full, X-Request-ID, Prometheus /metrics per device.
  • Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
  • Attention over long contexts (tested to 20,000 keys).

More: docs/serving.md (design and measurements).

Metadata

Release files for vkserve 0.1.16

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for vkserve 0.1.16
File Interpreter ABI Platform Actions
vkserve-0.1.16-py3-none-manylinux_2_34_x86_64.whl Python 3 none Linux glibc 2.34+ x86-64 Details for vkserve-0.1.16-py3-none-manylinux_2_34_x86_64.whl

Release files / vkserve-0.1.16-py3-none-manylinux_2_34_x86_64.whl

Download URL vkserve-0.1.16-py3-none-manylinux_2_34_x86_64.whl
Size 703.9 kB
Tags Linux glibc 2.34+ x86-64 Python 3
SHA-256 checksum
How to use checksums
dd40241447005ad45907506543e95e5b64953957ff9f948c7aaa53c8ee5bb9e2
BLAKE2b-256 checksum
How to use checksums
348fd33ea34ea29d2b05d9a4e9bfe5a6315afaf73e213ae44068f0f609f36ec7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for vkserve 0.1.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page