dstack 0.2.1

Easy-to-run ML workflows on any cloud

Easy-to-run ML workflows on any cloud

Define ML workflows as code and run via CLI. Use any cloud. Collaborate within teams.

Docs • Quick start • Usage • Slack

What is dstack?

dstack is an open-source tool that enables defining ML workflows as code, running them easily on any cloud while saving artifacts for reuse. It offers freedom to use any ML frameworks, cloud vendors, or third-party tools without requiring code changes.

Installation

Use pip to install dstack:

pip install dstack --upgrade

Configure a remote

To run workflows remotely (e.g. in a configured cloud account), configure a remote using the dstack config command.

dstack config

? Choose backend. Use arrows to move, type to filter
> [aws]
  [gcp]
  [hub]

If you intend to run remote workflows directly in the cloud using local cloud credentials, feel free to choose aws or gcp. Refer to AWS and GCP correspondingly for the details.

If you would like to manage cloud credentials, users and other settings centrally via a user interface, it is recommended to choose hub.

The hub remote is currently in an experimental phase. If you are interested in trying it out, please contact us via Slack.

Define workflows

Define ML workflows, their output artifacts, hardware requirements, and dependencies via YAML.

workflows:
  - name: mnist-data
    provider: bash
    commands:
      - pip install torchvision
      - python mnist/mnist_data.py
    artifacts:
      - path: ./data

  - name: train-mnist
    provider: bash
    deps:
      - workflow: mnist-data
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs

YAML eliminates the need to modify code in your scripts, giving you the freedom to choose frameworks, experiment trackers, and cloud providers.

Run workflows

Once a workflow is defined, you can use the dstack run command to run it either locally or remotely.

Run locally

By default, workflows run locally on your machine.

dstack run mnist-data

RUN        WORKFLOW    SUBMITTED  STATUS     TAG  BACKENDS
penguin-1  mnist-data  now        Submitted       local

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz

The artifacts from local workflows are also stored and can be reused in other local workflows.

Run remotely

To run a workflow remotely (e.g. in a configured cloud account), add the --remote flag to the dstack run command:

dstack run mnist-data --remote

RUN        WORKFLOW    SUBMITTED  STATUS     TAG  BACKENDS
mangust-1  mnist-data  now        Submitted       aws

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz

The output artifacts from remote workflows are also stored remotely and can be reused by other remote workflows.

The necessary hardware resources can be configured either via YAML or through arguments in the dstack run command, such as --gpu and --gpu-name.

dstack run train-mnist --remote --gpu 1

RUN       WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
turtle-1  train-mnist  now        Submitted       aws

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]
$ 

Upon running a workflow remotely, dstack automatically creates resources in the configured cloud account and destroys them once the workflow is complete.

More information

For additional information and examples, see the following links:

• Docs
• Installation
• Quick start
• Usage

Licence

Mozilla Public License 2.0