dstack 0.10.7rc3

dstack is an open-source tool that simplifies LLM development across multiple clouds.

Cost-effective LLM development

Docs • Examples • Blog • Slack

dstack is an open-source tool that simplifies LLM development across multiple clouds.

It streamlines development and deployment, reduces cloud costs, and frees users from vendor lock-in.

Latest news

• [2023/07] Port mapping, max duration, and more (Release)
• [2023/07] Serving with vLLM (Example)
• [2023/07] LLM as Chatbot (Example)
• [2023/07] Lambda Cloud GA and Docker support (Release)
• [2023/06] New YAML format (Release)

Installation

To use dstack, install it with pip, and start the server.

pip install "dstack[aws,gcp,azure,lambda]"
dstack start

On startup, the server sets up a default project that runs everything locally. To run dev environments and tasks in the cloud, log into the UI, create the corresponding project, and configure the CLI to use it.

Configurations

A configuration is a YAML file that describes what you want to run.

Note All configuration files must be named with the suffix .dstack.yml. For example, you can name the configuration file .dstack.yml or app.dstack.yml. You can define these configurations anywhere within your project.

Configurations can be of three types: dev-environment, task, and service.

Below is a configuration that runs a dev environment with a pre-built environment to which you can connect via VS Code Desktop.

type: dev-environment

init:
  - pip install -r requirements.txt

ide: vscode

Here's an example of a task configuration. A task can be either a batch job, such as training or fine-tuning a model, or a web application.

type: task

ports:
  - 7860

commands:
  - pip install -r requirements.txt
  - python app.py

A service configuration makes a web application available through the gateway. Below is a configuration for a simple web server.

type: service
gateway: my.domain.com
port: 80:8000
commands:
  - python -m http.server 8000

CLI

To run a configuration, use the dstack run command and pass the path to the directory with the configuration.

$ dstack run . 

 RUN          CONFIGURATION  USER   PROJECT  INSTANCE  RESOURCES        SPOT
 fast-moth-1  .dstack.yml    admin  local    -         5xCPUs, 15987MB  auto  

Starting SSH tunnel...

To open in VS Code Desktop, use this link:
  vscode://vscode-remote/ssh-remote+fast-moth-1/workflow

To exit, press Ctrl+C.

The CLI automatically provisions the required cloud resources and forwards the ports to your local machine. If you interrupt the run, the cloud resources will be released automatically.

Profiles

The .dstack/profiles.yml file allows to describe multiple profiles. ach profile can configure the project to use and the resources required for the run.

profiles:
  - name: gpu-large
    project: gcp
    resources:
       memory: 48GB
       gpu:
         memory: 24GB
    default: true

If you have configured the default profile, the dstack run command will use it automatically. Otherwise, you can always specify the profile using --profile PROFILE.

More information

For additional information and examples, see the following links:

• Docs
• Examples
• Blog
• Slack

Licence

Mozilla Public License 2.0