dstack 0.2rc1

Easy-to-run ML workflows on any cloud

Easy-to-run ML workflows on any cloud

Define ML workflows as code and run via CLI. Use any cloud. Collaborate within teams.

Docs • Quick start • Basics • Slack

dstack is the most easy way to define ML workflows as code and run them either locally or remotely on any cloud.

Highlighted features

• Define ML workflows declaratively as code
• Run workflows locally or remotely on any cloud (AWS, GCP, etc)
• Use on-demand on spot instances conveniently
• Save data, checkpoints, environments as artifacts and reuse them across workflows
• No need to use custom Docker images or Kubernetes

Installation

Use pip to install the dstack CLI:

pip install dstack --upgrade

Example

Here's an example from the Quick start.

workflows:
  - name: mnist-data
    provider: bash
    commands:
      - pip install torchvision
      - python mnist/mnist_data.py
    artifacts:
      - path: ./data

  - name: train-mnist
    provider: bash
    deps:
      - workflow: mnist-data
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - python mnist/train_mnist.py
    artifacts:
      - path: ./lightning_logs

YAML-defined workflows eliminate the need to modify code in your scripts, giving you the freedom to choose frameworks, experiment trackers, and cloud providers.

Run locally

Use the dstack CLI to run workflows locally:

dstack run mnist-data

Run remotely

To run workflows remotely (e.g. in the cloud) or share artifacts outside your machine, you must configure your remote settings using the dstack config command:

dstack config

This command will ask you to choose the type of backend (e.g. AWS), and the corresponding settings (e.g. the region where to run workflows, an S3 bucket where to store artifacts, etc).

Backend: aws
AWS profile: default
AWS region: eu-west-1
S3 bucket: dstack-142421590066-eu-west-1
EC2 subnet: none

For more details on how to configure a remote, check the installation guide.

Once a remote is configured, use the --remote flag with the dstack run command to run the workflow in the configured cloud:

dstack run mnist-data --remote

You can configure the required resources to run the workflows either via the resources property in YAML or the dstack run command's arguments, such as --gpu, --gpu-name, etc:

dstack run train-mnist --remote --gpu 1

When you run a workflow remotely, dstack automatically creates resources in the configured cloud, and releases them once the workflow is finished.

More information

For additional information and examples, see the following links:

• Docs
• Installation
• Quick start
• Basics

Licence

Mozilla Public License 2.0