dask-hpcconfig 2022.3.0

Configuration for various dask clusters

dask-hpcconfig

To install, use

python -m pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig

or clone the source:

git clone https://github.com/umr-lops/dask-hpcconfig.git
cd dask-hpcconfig

and then install from there:

python -m pip install .

or as "editable":

python -m pip install -e .

Usage

import dask_hpcconfig

To list the available cluster definitions:

dask_hpcconfig.print_clusters()

or, as a mapping of name to type:

clusters = dask_hpcconfig.available_clusters()

To create a cluster, use:

cluster = dask_hpcconfig.cluster(name)

where name is the name of one of the available clusters.

To override any particular setting: For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:

overrides = {"cluster.n_workers": 7}
cluster = dask_hpcconfig.cluster("datarmor-local", **overrides)

For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :

overrides = {"cluster.cores": 7}
cluster = dask_hpcconfig.cluster("datarmor", **overrides)
cluster.scale(49)

cluster can then be used to create a Client:

from distributed import Client

client = Client(cluster)