dstack 0.11.2rc1

dstack is an open-source toolkit for running LLM workloads across any clouds.

Run LLM workloads across any clouds

Docs • Examples • Blog • Slack

dstack is an open-source toolkit for running LLM workloads across any clouds, offering a cost-efficient and user-friendly interface for training, inference, and development.

Latest news ✨

• [2023/09] Managed gateways for serving LLMs across clouds (Release)
• [2023/08] Fine-tuning with Llama 2 (Example)
• [2023/08] Serving SDXL with FastAPI (Example)
• [2023/07] Serving LLMS with TGI (Example)
• [2023/07] Serving LLMS with vLLM (Example)

Installation

To use dstack, install it with pip, and start the server.

pip install "dstack[aws,gcp,azure,lambda]"
dstack start

Configure backends

Upon startup, the server sets up the default project called main. Prior to using dstack, you must log in to the UI, open the project's settings, and configure cloud backends (e.g., AWS, GCP, Azure, Lambda, etc.).

Define a configuration

A configuration is a YAML file that describes what you want to run.

Note All configuration files must be named with the suffix .dstack.yml. For example, you can name the configuration file .dstack.yml or app.dstack.yml. You can define these configurations anywhere within your project.

Configurations can be of three types: dev-environment, task, and service.

Dev environments

A dev environment is a virtual machine with a pre-configured IDE.

type: dev-environment

python: "3.11" # (Optional) If not specified, your local version is used

setup: # (Optional) Executed once at the first startup
  - pip install -r requirements.txt

ide: vscode

Tasks

A task can be either a batch job, such as training or fine-tuning a model, or a web application.

type: task

python: "3.11" # (Optional) If not specified, your local version is used

ports:
  - 7860

commands:
  - pip install -r requirements.txt
  - python app.py

While the task runs in the cloud, the CLI forwards traffic, allowing you to access the application from your local machine.

Services

A service is an application that is accessible through a public endpoint.

type: service

port: 7860

commands:
  - pip install -r requirements.txt
  - python app.py

Once the service is up, dstack makes it accessible from the Internet through the gateway.

CLI

To run a configuration, use the dstack run command followed by working directory and the path to the configuration file.

dstack run . -f text-generation-inference/serve.dstack.yml --gpu A100 -y

 RUN           BACKEND  INSTANCE              SPOT  PRICE STATUS    SUBMITTED
 tasty-zebra-1 lambda   200GB, 1xA100 (80GB)  no    $1.1  Submitted now
 
Privisioning...

Serving on https://tasty-zebra-1.mydomain.com

dstack automatically provisions cloud resources based in the configured clouds that offer the best price and availability.

More information

For additional information and examples, see the following links:

• Docs
• Examples
• Blog
• Slack

Licence

Mozilla Public License 2.0