gpustack 0.3.1

GPUStack

GPUStack is an open-source GPU cluster manager for running large language models(LLMs).

Key Features

• Supports a Wide Variety of Hardware: Run with different brands of GPUs in Apple MacBooks, Windows PCs, and Linux servers.
• Scales with Your GPU Inventory: Easily add more GPUs or nodes to scale up your operations.
• Distributed Inference: Supports both single-node multi-GPU and multi-node inference and serving.
• Multiple Inference Backends: Supports llama-box (llama.cpp) and vLLM as the inference backends.
• Lightweight Python Package: Minimal dependencies and operational overhead.
• OpenAI-compatible APIs: Serve APIs that are compatible with OpenAI standards.
• User and API key management: Simplified management of users and API keys.
• GPU metrics monitoring: Monitor GPU performance and utilization in real-time.
• Token usage and rate metrics: Track token usage and manage rate limits effectively.

Installation

Linux or MacOS

GPUStack provides a script to install it as a service on systemd or launchd based systems. To install GPUStack using this method, just run:

curl -sfL https://get.gpustack.ai | sh -s -

Optionally, you can add extra workers to form a GPUStack cluster by running the following command on other nodes (replace http://myserver and mytoken with your actual server URL and token):

curl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken

In the default setup, you can run the following to get the token used for adding workers:

cat /var/lib/gpustack/token

Windows

Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:

Invoke-Expression (Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content

Optionally, you can add extra workers to form a GPUStack cluster by running the following command on other nodes (replace http://myserver and mytoken with your actual server URL and token):

Invoke-Expression "& { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } --server-url http://myserver --token mytoken"

In the default setup, you can run the following to get the token used for adding workers:

Get-Content -Path "$env:APPDATA\gpustack\token" -Raw

Manual Installation

For manual installation or detailed configurations, refer to the installation docs.

Getting Started

1. Run and chat with the llama3.2 model:

gpustack chat llama3.2 "tell me a joke."

2. Open http://myserver in the browser to access the GPUStack UI. Log in to GPUStack with username admin and the default password. You can run the following command to get the password for the default setup:

Linux or MacOS

cat /var/lib/gpustack/initial_admin_password

Windows

Get-Content -Path "$env:APPDATA\gpustack\initial_admin_password" -Raw

3. Click Playground in the navigation menu. Now you can chat with the LLM in the UI playground.

4. Click API Keys in the navigation menu, then click the New API Key button.

5. Fill in the Name and click the Save button.

6. Copy the generated API key and save it somewhere safe. Please note that you can only see it once on creation.

7. Now you can use the API key to access the OpenAI-compatible API. For example, use curl as the following:

export GPUSTACK_API_KEY=myapikey
curl http://myserver/v1-openai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GPUSTACK_API_KEY" \
  -d '{
    "model": "llama3.2",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "stream": true
  }'

Supported Platforms

• MacOS
• Linux
• Windows

Supported Accelerators

• Apple Metal
• NVIDIA CUDA

We plan to support the following accelerators in future releases.

• AMD ROCm
• Intel oneAPI
• MTHREADS MUSA
• Qualcomm AI Engine

Supported Models

GPUStack uses llama.cpp and vLLM as the backends and supports a wide range of models. Models from the following sources are supported:

1. Hugging Face

2. ModelScope

3. Ollama Library

Example language models:

• LLaMA
• Mistral 7B
• Mixtral MoE
• Falcon
• Baichuan
• Yi
• Deepseek
• Qwen
• Phi
• Grok-1

Example multimodal models:

• Llama3.2-Vision
• Pixtral
• Qwen2-VL
• LLaVA
• InternVL2

For full list of supported models, please refer to the supported models section in the inference backends documentation.

OpenAI-Compatible APIs

GPUStack serves the following OpenAI compatible APIs under the /v1-openai path:

• List Models
• Create Completions
• Create Chat Completions
• Create Embeddings

For example, you can use the official OpenAI Python API library to consume the APIs:

from openai import OpenAI
client = OpenAI(base_url="http://myserver/v1-openai", api_key="myapikey")

completion = client.chat.completions.create(
  model="llama3.2",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

GPUStack users can generate their own API keys in the UI.

Documentation

Please see the official docs site for complete documentation.

Build

1. Install python 3.10+.

2. Run make build.

You can find the built wheel package in dist directory.

Contributing

Please read the Contributing Guide if you're interested in contributing to GPUStack.

License

Copyright (c) 2024 The GPUStack authors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at LICENSE file for details.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.