Skip to main content

Use dask to run the DVC graph

Project description

Coverage Status PyTest PyPI version zincware

Dask4DVC - Distributed Node Exectuion

DVC provides tools for building and executing the computational graph locally through various methods. The dask4dvc package combines Dask Distributed with DVC to make it easier to use with HPC managers like Slurm.

The dask4dvc repro package will run the DVC graph in parallel where possible. Currently, dask4dvc run will not run stages per experiment sequentially.

:warning: This is an experimental package not affiliated in any way with iterative or DVC.

Usage

Dask4DVC provides a CLI similar to DVC.

  • dvc repro becomes dask4dvc repro.
  • dvc queue start becomes dask4dvc run

You can follow the progress using dask4dvc <cmd> --dashboard.

SLURM Cluster

You can use dask4dvc easily with a slurm cluster. This requires a running dask scheduler:

from dask_jobqueue import SLURMCluster

cluster = SLURMCluster(
    cores=1, memory='128GB',
    queue="gpu",
    processes=1,
    walltime='8:00:00',
    job_cpu=1,
    job_extra=['-N 1', '--cpus-per-task=1', '--tasks-per-node=64', "--gres=gpu:1"],
    scheduler_options={"port": 31415}
)
cluster.adapt()

with this setup you can then run dask4dvc repro --address 127.0.0.1:31415 on the example port 31415.

You can also use config files with dask4dvc repro --config myconfig.yaml. All dask.distributed Clusters should be supported.

default:
  SGECluster:
    queue: regular
    cores: 10
    memory: 16 GB

dask4dvc repro

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask4dvc-0.2.3.tar.gz (11.2 kB view details)

Uploaded Source

Built Distribution

dask4dvc-0.2.3-py3-none-any.whl (12.8 kB view details)

Uploaded Python 3

File details

Details for the file dask4dvc-0.2.3.tar.gz.

File metadata

  • Download URL: dask4dvc-0.2.3.tar.gz
  • Upload date:
  • Size: 11.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.10.8 Linux/5.19.0-40-generic

File hashes

Hashes for dask4dvc-0.2.3.tar.gz
Algorithm Hash digest
SHA256 de696c0c9e79f5583a4352434bee41f321113bb19f8a7303fa3627a82bc3accb
MD5 213e2c010bfddc9491583f0512e13472
BLAKE2b-256 f7b618163d26a00668f314f1d3c3146ec93fa9f1fe78e0253ae0cad0216c4a3a

See more details on using hashes here.

File details

Details for the file dask4dvc-0.2.3-py3-none-any.whl.

File metadata

  • Download URL: dask4dvc-0.2.3-py3-none-any.whl
  • Upload date:
  • Size: 12.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.10.8 Linux/5.19.0-40-generic

File hashes

Hashes for dask4dvc-0.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 e5adf2f493794d8f5750d32ce3ed834859d2826491d3c18c987b267b553ddc82
MD5 cb7a3a80f311e9537bd9ba651debf1b7
BLAKE2b-256 17860e1d09f8e95893fe21b2715eaccbbc287baa7650dc7fe079172827d136ae

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page