GPUStack Runner

This repository serves as the Docker image pack center for GPUStack Runner. It provides a collection of Dockerfiles to build images for various inference services across different accelerated backends.

Agenda

Onboard Services
Directory Structure
Dockerfile Convention
Docker Image Naming Convention
Integration Process

Onboard Services

[!TIP]

The list below shows the accelerated backends and inference services available in the latest release. For support of backends or services not shown here, please refer to previous release tags.

Deprecated inference service versions in the latest release are marked with ~~strikethrough~~ formatting. They may still be available in previous releases, and not recommended for new deployments.

Polished inference service versions in the latest release are marked with bold formatting. If they are using in your deployment, it is recommended to pull the latest images and upgrade.

The following table lists the supported accelerated backends and their corresponding inference services with versions.

Ascend CANN

CANN Version (Variant)	MindIE	vLLM	SGLang
9.1 (950/A5)		`0.23.0`
9.1 (A3/910C)		`0.23.0`
9.1 (910B)		`0.23.0`
9.1 (310P)		`0.23.0`
9.0 (A3/910C)		`0.20.2`(rc)	`0.5.18`, `0.5.15.post1`, `0.5.14`
9.0 (910B)		`0.20.2`(rc)	`0.5.18`, `0.5.15.post1`, `0.5.14`
9.0 (310P)		`0.20.2`(rc)
8.5 (A3/910C)	`2.3.0`	`0.18.0`, `0.17.0`(rc), `0.16.0`(rc), `0.15.0`(rc), `0.14.1`(rc), `0.13.0`	`0.5.12.post1`, `0.5.9`, `0.5.8.post1`
8.5 (910B)	`2.3.0`	`0.18.0`, `0.17.0`(rc), `0.16.0`(rc), `0.15.0`(rc), `0.14.1`(rc), `0.13.0`	`0.5.12.post1`, `0.5.9`, `0.5.8.post1`
8.5 (310P)	`2.3.0`	`0.18.0`, `0.17.0`(rc), `0.16.0`(rc), `0.15.0`(rc), `0.14.1`(rc)
8.3 (A3/910C)	`2.2.rc1`	`0.12.0`(rc), `0.11.0`	`0.5.7`, `0.5.6.post2`
8.3 (910B)	`2.2.rc1`	`0.12.0`(rc), `0.11.0`	`0.5.7`, `0.5.6.post2`
8.3 (310P)	`2.2.rc1`
8.2 (A3/910C)	`2.1.rc2`	`0.10.2`(rc)
8.2 (910B)	`2.1.rc2`	`0.10.2`(rc), `0.10.0`(rc), `0.9.1`
8.2 (310P)	`2.1.rc2`	`0.10.0`(rc), `0.9.1`

Iluvatar CoreX

CoreX Version (Variant)	vLLM
4.2	`0.8.3`

NVIDIA CUDA

[!NOTE]

CUDA 13.0 supports Compute Capabilities: 7.5 8.0+PTX 8.9 9.0 10.0 10.3 12.0+PTX.

CUDA 12.9 supports Compute Capabilities: 7.5 8.0+PTX 8.9 9.0 10.0 10.3 12.0 12.1+PTX.

CUDA 12.8 supports Compute Capabilities: 7.5 8.0+PTX 8.9 9.0 10.0+PTX 12.0+PTX.

CUDA 12.6/12.4 supports Compute Capabilities: 7.5 8.0+PTX 8.9 9.0+PTX.

CUDA Version (Variant)	vLLM	SGLang	VoxBox
13.0	`0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, `0.21.0`, `0.20.2`, `0.19.1`, `0.18.1`	`0.5.18`, `0.5.15.post1`, `0.5.14`, `0.5.12.post1`
12.9	`0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, `0.21.0`, `0.20.2`, `0.19.1`, `0.18.1`, `0.17.1`, `0.16.0`, `0.15.1`, `0.14.1`, `0.13.0`, `0.12.0`, `0.11.2`	`0.5.18`, `0.5.15.post1`, `0.5.14`, `0.5.12.post1`, `0.5.9`, `0.5.8.post1`, `0.5.7`, `0.5.6.post2`
12.8	`0.17.1`, `0.16.0`, `0.15.1`, `0.14.1`, `0.13.0`, `0.12.0`, `0.11.2`, `0.10.2`	`0.5.9`, `0.5.8.post1`, `0.5.7`, `0.5.6.post2`, `0.5.5.post3`	`0.0.21`
12.6	`0.15.1`, `0.14.1`, `0.13.0`, `0.12.0`, `0.11.2`, `0.10.2`		`0.0.21`

Hygon DTK

DTK Version (Variant)	vLLM	SGLang
26.04	`0.18.1`	`0.5.10`(rc)
25.04	`0.18.1`, `0.11.0`, `0.9.2`, `0.8.5`

T-Head HGGC

HGGC Version (Variant)	vLLM	SGLang
13.0	`0.23.0`, `0.20.1`, `0.19.0`, `0.18.0`	`0.5.12`, `0.5.10`, `0.5.9`
12.3	`0.12.0`, `0.11.1`	`0.5.6`, `0.5.5`

MetaX MACA

MACA Version (Variant)	vLLM	SGLang
3.7	`0.21.0`, `0.20.0`	`0.5.11`, `0.5.10`
3.5	`0.14.0`	`0.5.9`
3.3	`0.11.2`	`0.5.6`
3.2	`0.10.2`
3.0	`0.9.1`

MThreads MUSA

MUSA Version (Variant)	vLLM	SGLang
4.3.2		`0.5.7`
4.1.0	`0.9.2`

AMD ROCm

[!NOTE]

ROCm 7.1/7.0 supports LLVM targets: gfx908 gfx90a gfx942 gfx950 gfx1030 gfx1100 gfx1101 gfx1200 gfx1201 gfx1150 gfx1151.

ROCm 6.4 supports LLVM targets: gfx908 gfx90a gfx942 gfx1030 gfx1100.

[!WARNING]

ROCm 7.0 vLLM 0.11.2 are reusing the official ROCm 6.4 PyTorch 2.9 wheel package rather than a ROCm 7.0 specific PyTorch build. Although supports ROCm 7.0 in vLLM 0.11.2, gfx1150/gfx1151 are not supported yet.

ROCm 6.4 vLLM 0.13.0 supports gfx903 gfx90a gfx942 only.

ROCm 6.4 SGLang supports gfx942 only.

ROCm 7.0 SGLang supports gfx950 only.

ROCm Version (Variant)	vLLM	SGLang
7.2	`0.27.1`, `0.25.1`, `0.24.0`, `0.22.1`, `0.21.0`, `0.20.2`, `0.19.1`	`0.5.18`, `0.5.15.post1`, `0.5.14`, `0.5.12.post1`
7.1	`0.17.1`
7.0	`0.18.1`, `0.16.0`, `0.15.1`, `0.14.1`, `0.13.0`, `0.12.0`, `0.11.2`	`0.5.9`, `0.5.8.post1`, `0.5.7`, `0.5.6.post2`
6.4	`0.16.0`, `0.15.1`, `0.14.1`, `0.13.0`, `0.12.0`, `0.11.2`, `0.10.2`	`0.5.8.post1`, `0.5.7`, `0.5.6.post2`, `0.5.5.post3`

Directory Structure

The pack skeleton is organized by backend:

pack
├── {BACKEND 1}
│   └── Dockerfile
├── {BACKEND 2}
│   └── Dockerfile
├── {BACKEND 3}
│   └── Dockerfile
├── ...
│   └── Dockerfile
└── {BACKEND N}
    └── Dockerfile

Dockerfile Convention

Each Dockerfile follows these conventions:

Begin with comments describing the package logic in steps and usage of build arguments (ARGs).
Use ARG for all required and optional build arguments. If a required argument is unused, mark it as (PLACEHOLDER).
Use heredoc syntax for RUN commands to improve readability.

Example Dockerfile Structure

# Describe package logic and ARG usage.
#
ARG PYTHON_VERSION=...                                 # REQUIRED
ARG CMAKE_MAX_JOBS=...                                 # REQUIRED
ARG {OTHERS}                                           # OPTIONAL
ARG {BACKEND}_VERSION=...                              # REQUIRED
ARG {BACKEND}_VERSION_EXTRA=...                        # OPTIONAL
ARG {BACKEND}_ARCHS=...                                # REQUIRED
ARG {BACKEND}_{OTHERS}=...                             # OPTIONAL
ARG {SERVICE}_BASE_IMAGE=...                           # REQUIRED
ARG {SERVICE}_VERSION=...                              # REQUIRED
ARG {SERVICE}_{OTHERS}=...                             # OPTIONAL
ARG {SERVICE}_{FRAMEWORK}_VERSION=...                  # REQUIRED
ARG {SERVICE}_{FRAMEWORK}_{OTHERS}=...                 # OPTIONAL

# Stage Bake Runtime
FROM {BACKEND DEVEL IMAGE} AS runtime
SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
ARG TARGETPLATFORM
ARG TARGETOS
ARG TARGETARCH
ARG ...
RUN <<EOF
    # TODO: install runtime dependencies
EOF

# Stage Install Service
FROM {BACKEND}_BASE_IMAGE AS {service}
SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
ARG TARGETPLATFORM
ARG TARGETOS
ARG TARGETARCH
ARG ...
RUN <<EOF
    # TODO: install service and dependencies
EOF

WORKDIR /
ENTRYPOINT [ "tini", "--" ]

Docker Image Naming Convention

The Docker image naming convention is as follows:

Multi-architecture image names: {NAMESPACE}/{REPOSITORY}:{TAG}.
Single-architecture image tags: {BACKEND}{BACKEND_VERSION%.*}[-{BACKEND_VARIANT}]-{SERVICE}{SERVICE_VERSION}-{OS}-{ARCH}.
Multi-architecture image tags: {BACKEND}{BACKEND_VERSION%.*}[-{BACKEND_VARIANT}]-{SERVICE}{SERVICE_VERSION}[-dev].
All names adn tags must be lowercase.

Example

NAMESPACE: gpustack
REPOSITORY: runner

Accelerated Backend	OS/ARCH	Inference Service	Single-Arch Image Name	Multi-Arch Image Name
Ascend CANN 910b	linux/amd64	vLLM	`gpustack/runner:cann8.1-910b-vllm0.9.2-linux-amd64`	`gpustack/runner:cann8.1-910b-vllm0.9.2`
Ascend CANN 910b	linux/arm64	vLLM	`gpustack/runner:cann8.1-910b-vllm0.9.2-linux-arm64`	`gpustack/runner:cann8.1-910b-vllm0.9.2`
NVIDIA CUDA 12.8	linux/amd64	vLLM	`gpustack/runner:cuda12.8-910b-vllm0.9.2-linux-amd64`	`gpustack/runner:cuda12.8-910b-vllm0.9.2`
NVIDIA CUDA 12.8	linux/arm64	vLLM	`gpustack/runner:cuda12.8-910b-vllm0.9.2-linux-arm64`	`gpustack/runner:cuda12.8-910b-vllm0.9.2`

Build and Release Workflow

Build single architecture images for OS/ARCH, e.g. gpustack/runner:cann8.1-910b-vllm0.9.2-linux-amd64.
Combine single-architecture images into a multiple architectures image, e.g. gpustack/runner:cann8.1-910b-vllm0.9.2-dev.
After testing, rename the multi-architecture image to the final tag, e.g. gpustack/runner:cann8.1-910b-vllm0.9.2.

Integration Process

Ingesting a New Accelerated Backend

To add support for a new accelerated backend:

Create a new directory under pack/ named with the new backend.
Add a Dockerfile in the new directory following the Dockerfile Convention.
Update pack.yml, discard.yml and prune.yml to include the new backend in the build matrix.
Update matrix.yml to include the new backend and its variants.
Update _RE_DOCKER_IMAGE in runner.py to recognize the new backend.
[Optional] Update tests if necessary.

Ingesting a New Inference Service

To add support for a new inference service:

Modify the Dockerfile of the relevant backend in pack/{BACKEND}/Dockerfile to include the new service.
Update pack.yml to include the new service in the build matrix.
Update matrix.yml to include the new service.
Update _RE_DOCKER_IMAGE in runner.py to recognize the new service.
[Optional] Update tests if necessary.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at LICENSE file for details.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Metadata

Release files for gpustack-runner 0.1.29

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gpustack-runner 0.1.29
File	Size	Uploaded
gpustack_runner-0.1.29.tar.gz	13.5 MB	Sep 1, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for gpustack-runner 0.1.29
File	Interpreter	ABI	Platform	Reset
gpustack_runner-0.1.29-py3-none-any.whl 30.9 kB Sep 1, 2026	Python 3	none	any	Details

Total release size: 13.5 MB

Release files / gpustack_runner-0.1.29.tar.gz

Download URL	gpustack_runner-0.1.29.tar.gz
Size	13.5 MB
Tags	Source
SHA-256 checksum How to use checksums	`22c4ec21c3aee2c9d48f3535397821c85172370654ad71549d09289ca00ec393`
BLAKE2b-256 checksum How to use checksums	`7b5bfcd80ec9f3b74d36c3e5e1462af5d29604585a3050d9bb6d36d50f6d4397`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.8.24`

Release files / gpustack_runner-0.1.29-py3-none-any.whl

Download URL	gpustack_runner-0.1.29-py3-none-any.whl
Size	30.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a0330f9512a4ebd0261acba41819e19e89188b47c466ca31db8499fbf9843686`
BLAKE2b-256 checksum How to use checksums	`dd41baf89298ead365239a326583b7310500cb05eda6b0113aade845181995d3`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.8.24`