dstack 0.11.1

dstack is an open-source toolkit for running LLM workloads across any clouds.

Run LLM workloads across any clouds

Docs • Examples • Blog • Slack

dstack is an open-source toolkit for running LLM workloads across any clouds, offering a cost-efficient and user-friendly interface for training, inference, and development.

Latest news ✨

• [2023/08] Fine-tuning with Llama 2 (Example)
• [2023/08] An early preview of services (Release)
• [2023/08] Serving SDXL with FastAPI (Example)
• [2023/07] Serving LLMS with TGI (Example)
• [2023/07] Serving LLMS with vLLM (Example)

Installation

To use dstack, install it with pip, and start the server.

pip install "dstack[aws,gcp,azure,lambda]"
dstack start

Configure backends

Upon startup, the server sets up the default project called main. Prior to using dstack, you must log in to the UI, open the project's settings, and configure cloud backends (e.g., AWS, GCP, Azure, Lambda, etc.).

Define a configuration

A configuration is a YAML file that describes what you want to run.

Note All configuration files must be named with the suffix .dstack.yml. For example, you can name the configuration file .dstack.yml or app.dstack.yml. You can define these configurations anywhere within your project.

Configurations can be of three types: dev-environment, task, and service.

Dev environments

A dev environment is a virtual machine with a pre-configured IDE.

type: dev-environment

python: "3.11" # (Optional) If not specified, your local version is used

setup: # (Optional) Executed once at the first startup
  - pip install -r requirements.txt

ide: vscode

Tasks

A task can be either a batch job, such as training or fine-tuning a model, or a web application.

type: task

python: "3.11" # (Optional) If not specified, your local version is used

ports:
  - 7860

commands:
  - pip install -r requirements.txt
  - python app.py

While the task runs in the cloud, the CLI forwards traffic, allowing you to access the application from your local machine.

Services

A service is an application that is accessible through a public endpoint.

type: service

port: 7860

commands:
  - pip install -r requirements.txt
  - python app.py

Once the service is up, dstack makes it accessible from the Internet through the gateway.

CLI

To run a configuration, use the dstack run command followed by working directory and the path to the configuration file.

dstack run . -f serve.dstack.yml

dstack automatically provisions cloud resources based in the configured clouds that offer the best price and availability.

For every run, you can specify hardware resources like memory and GPU, along with various run policies (e.g., maximum hourly price, use of spot instances, etc.).

Example	Description
dstack run . --gpu A10	Use an instance with NVIDIA A10 GPU
dstack run . --gpu A100:8	Use an instance with 8 NVIDIA A100 GPUs
dstack run . --gpu 24GB	Use an instance with a GPU that has 24GB

More information

For additional information and examples, see the following links:

• Docs
• Examples
• Blog
• Slack

Licence

Mozilla Public License 2.0