Skip to main content

Configuration for various dask clusters

Project description

dask-hpcconfig

To install, use

python -m pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig

or clone the source:

git clone https://github.com/umr-lops/dask-hpcconfig.git
cd dask-hpcconfig

and then install from there:

python -m pip install .

or as "editable":

python -m pip install -e .

Usage

import dask_hpcconfig

To list the available cluster definitions:

dask_hpcconfig.print_clusters()

or, as a mapping of name to type:

clusters = dask_hpcconfig.available_clusters()

To create a cluster, use:

cluster = dask_hpcconfig.cluster(name)

where name is the name of one of the available clusters.

To override any particular setting: For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:

overrides = {"cluster.n_workers": 7}
cluster = dask_hpcconfig.cluster("datarmor-local", **overrides)

For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :

overrides = {"cluster.cores": 7}
cluster = dask_hpcconfig.cluster("datarmor", **overrides)
cluster.scale(49)

cluster can then be used to create a Client:

from distributed import Client

client = Client(cluster)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-hpcconfig-2022.3.0.tar.gz (11.2 kB view hashes)

Uploaded Source

Built Distribution

dask_hpcconfig-2022.3.0-py3-none-any.whl (7.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page