dstack 0.7.2

The hassle-free tool for managing ML workflows on any cloud platform.

Automate your ML workflows on any cloud

The hassle-free tool for managing ML workflows on any cloud platform.

Docs • Quick start • Playground • Setup • Usage • Examples

What is dstack?

dstack is an open-source tool that automates ML workflows, enabling effective management on any cloud platform.

It empowers your team to prepare data, train, and fine-tune models using their preferred frameworks and dev environments without spending time on engineering and infrastructure.

Install the CLI

Use pip to install dstack:

pip install dstack

Configure a remote

By default, workflows run locally. To run workflows remotely (e.g. in a configured cloud account), configure a remote using the dstack config command.

dstack config

? Choose backend. Use arrows to move, type to filter
> [aws]
  [gcp]
  [hub]

Choose hub if you prefer managing cloud credentials and settings through a user interface while working in a team.

For running remote workflows with local cloud credentials, select aws or gcp.

Define workflows

Define ML workflows, their output artifacts, hardware requirements, and dependencies via YAML.

workflows:
  - name: train-mnist
    provider: bash
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python examples/mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs

Run locally

By default, workflows run locally on your machine.

dstack run train-mnist

RUN        WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
penguin-1  train-mnist  now        Submitted       local

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

Run remotely

To run a workflow remotely (e.g. in a configured cloud account), add the --remote flag to the dstack run command:

The necessary hardware resources can be configured either via YAML or through arguments in the dstack run command, such as --gpu and --gpu-name.

dstack run train-mnist --remote --gpu 1

RUN       WORKFLOW     SUBMITTED  STATUS     TAG  BACKENDS
turtle-1  train-mnist  now        Submitted       aws

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

GPU available: True, used: True

Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]

Upon running a workflow remotely, dstack automatically creates resources in the configured cloud account and destroys them once the workflow is complete.

Providers

dstack supports multiple providers to set up environments, run scripts, and launch interactive development environments and applications.

Artifacts

dstack allows you to save output artifacts and conveniently reuse them across workflows.

More information

For additional information and examples, see the following links:

• Docs
• Quick start
• Playground
• Setup
• Usage
• Examples

Licence

Mozilla Public License 2.0