dstack 0.8.1

The easiest way to define ML workflows and run them on any cloud platform

ML workflows as code

The easiest way to run ML workflows on any cloud platform

Quick start • Docs • Tutorials • Blog

What is dstack?

dstack makes it very easy to define ML workflows and run them on any cloud platform. It provisions infrastructure, manages data, and monitors usage for you.

Ideal for processing data, training models, running apps, and any other ML development tasks.

Installation and setup

To use dstack, install it with pip and start the Hub application.

pip install dstack
dstack start

The dstack start command starts the Hub application, and creates the default project to run workflows locally.

If you'll want to run workflows in the cloud (e.g. AWS, or GCP), simply log into the Hub application, and create a new project.

Run your first workflows

Let's define our first ML workflow in .dstack/workflows/hello.yaml:

workflows:
  - name: train-mnist
    provider: bash
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python examples/mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs

The YAML file allows you to request hardware resources, run Python, save artifacts, use cache and
dependencies, create dev environments, run apps, and many more.

Run it

Go ahead and run it:

dstack run train-mnist

RUN        WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
penguin-1  train-mnist  now        Submitted       local

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: False, used: False

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

The dstack run command runs the workflow using the settings specified for the project configured with the Hub application.

Create a Hub project

As mentioned above, the default project runs workflows locally. However, you can log into the application and create other projects that allow you to run workflows in the cloud.

If you want the project to use the cloud, you'll need to provide cloud credentials and specify settings such as the artifact storage bucket and the region where the workflows will run.

Once a project is created, copy the CLI command from the project settings and execute it in your terminal.

dstack config --url http://127.0.0.1:3000 \
  --project gcp \
  --token b934d226-e24a-4eab-a284-eb92b353b10f

The dstack config command configures dstack to run workflows using the settings from the corresponding project.

You can configure multiple projects and use them interchangeably (by passing the --project argument to the dstack run command. Any project can be set as the default by passing --default to the dstack config command.

Configuring multiple projects can be convenient if you want to run workflows both locally and in the cloud or if you would like to use multiple clouds.

Manage resources

Consider that you have configured a project that allows you to use a GPU (e.g., a local backend if you have a GPU locally, or an AWS or GCP backend).

Let's update our workflow and add resources.

workflows:
  - name: train-mnist
    provider: bash
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python examples/mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs
    resources:
      gpu:
        name: V100
        count: 1

Let's run the workflow:

dstack run train-mnist --project gcp

RUN        WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
penguin-1  train-mnist  now        Submitted       local

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

If your project is configured to use the cloud, the Hub application will automatically create the necessary cloud resources to execute the workflow and tear them down once it is finished.

More information

For additional information and examples, see the following links:

• Quick start
• Docs
• Tutorials
• Blog

Licence

Mozilla Public License 2.0