dstack is an open-source toolkit for running LLM workloads across any clouds.
Project description
dstack
is an open-source toolkit for running LLM workloads across any clouds, offering a
cost-efficient and user-friendly interface for training, inference, and development.
Latest news ✨
- [2023/09] Managed gateways for serving LLMs across clouds (Release)
- [2023/08] Fine-tuning with Llama 2 (Example)
- [2023/08] Serving SDXL with FastAPI (Example)
- [2023/07] Serving LLMS with TGI (Example)
- [2023/07] Serving LLMS with vLLM (Example)
Installation
To use dstack
, install it with pip
, and start the server.
pip install "dstack[aws,gcp,azure,lambda]"
dstack start
Configure backends
Upon startup, the server sets up the default project called main
. Prior to using dstack
, you must log in to the
UI, open the project's settings, and configure cloud backends
(e.g., AWS, GCP, Azure,
Lambda, etc.).
Define a configuration
A configuration is a YAML file that describes what you want to run.
Note All configuration files must be named with the suffix
.dstack.yml
. For example, you can name the configuration file.dstack.yml
orapp.dstack.yml
. You can define these configurations anywhere within your project.
Configurations can be of three types: dev-environment
, task
, and service
.
Dev environments
A dev environment is a virtual machine with a pre-configured IDE.
type: dev-environment
python: "3.11" # (Optional) If not specified, your local version is used
setup: # (Optional) Executed once at the first startup
- pip install -r requirements.txt
ide: vscode
Tasks
A task can be either a batch job, such as training or fine-tuning a model, or a web application.
type: task
python: "3.11" # (Optional) If not specified, your local version is used
ports:
- 7860
commands:
- pip install -r requirements.txt
- python app.py
While the task runs in the cloud, the CLI forwards traffic, allowing you to access the application from your local machine.
Services
A service is an application that is accessible through a public endpoint.
type: service
port: 7860
commands:
- pip install -r requirements.txt
- python app.py
Once the service is up, dstack
makes it accessible from the Internet through
the gateway.
CLI
To run a configuration, use the dstack run
command followed by
working directory and the path to the configuration file.
dstack run . -f text-generation-inference/serve.dstack.yml --gpu A100 -y
RUN BACKEND INSTANCE SPOT PRICE STATUS SUBMITTED
tasty-zebra-1 lambda 200GB, 1xA100 (80GB) no $1.1 Submitted now
Privisioning...
Serving on https://tasty-zebra-1.mydomain.com
dstack
automatically provisions cloud resources based in the
configured clouds that offer the best price and availability.
More information
For additional information and examples, see the following links:
Licence
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dstack-0.11.2rc2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca4be76e62b281bfef11cdaa6c13704afdff92c9baa474650e2d1a6776c74c88 |
|
MD5 | 97869105ed196eaaa9117d3a06bffaeb |
|
BLAKE2b-256 | 4e6ccfff999bff04528807f0db0a6f8a3e6131788c8707325ec5e98b0445702b |