Use dask to run the DVC graph
Project description
Dask4DVC - Distributed Node Exectuion
DVC provides tools for building and executing the computational graph locally through various methods.
The dask4dvc
package combines Dask Distributed with DVC to make it easier to use with HPC managers like Slurm.
The dask4dvc repro
package will run the DVC graph in parallel where possible.
Currently, dask4dvc run
will not run stages per experiment sequentially.
:warning: This is an experimental package not affiliated in any way with iterative or DVC.
Usage
Dask4DVC provides a CLI similar to DVC.
dvc repro
becomesdask4dvc repro
.dvc queue start
becomesdask4dvc run
You can follow the progress using dask4dvc <cmd> --dashboard
.
SLURM Cluster
You can use dask4dvc
easily with a slurm cluster.
This requires a running dask scheduler:
from dask_jobqueue import SLURMCluster
cluster = SLURMCluster(
cores=1, memory='128GB',
queue="gpu",
processes=1,
walltime='8:00:00',
job_cpu=1,
job_extra=['-N 1', '--cpus-per-task=1', '--tasks-per-node=64', "--gres=gpu:1"],
scheduler_options={"port": 31415}
)
cluster.adapt()
with this setup you can then run dask4dvc repro --address 127.0.0.1:31415
on the example port 31415
.
You can also use config files with dask4dvc repro --config myconfig.yaml
.
All dask.distributed
Clusters should be supported.
default:
SGECluster:
queue: regular
cores: 10
memory: 16 GB
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dask4dvc-0.2.3.tar.gz
.
File metadata
- Download URL: dask4dvc-0.2.3.tar.gz
- Upload date:
- Size: 11.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.10.8 Linux/5.19.0-40-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | de696c0c9e79f5583a4352434bee41f321113bb19f8a7303fa3627a82bc3accb |
|
MD5 | 213e2c010bfddc9491583f0512e13472 |
|
BLAKE2b-256 | f7b618163d26a00668f314f1d3c3146ec93fa9f1fe78e0253ae0cad0216c4a3a |
File details
Details for the file dask4dvc-0.2.3-py3-none-any.whl
.
File metadata
- Download URL: dask4dvc-0.2.3-py3-none-any.whl
- Upload date:
- Size: 12.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.10.8 Linux/5.19.0-40-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5adf2f493794d8f5750d32ce3ed834859d2826491d3c18c987b267b553ddc82 |
|
MD5 | cb7a3a80f311e9537bd9ba651debf1b7 |
|
BLAKE2b-256 | 17860e1d09f8e95893fe21b2715eaccbbc287baa7650dc7fe079172827d136ae |