Configuration for various dask clusters
Project description
dask-hpcconfig
To install, use
python -m pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig
or clone the source:
git clone https://github.com/umr-lops/dask-hpcconfig.git
cd dask-hpcconfig
and then install from there:
python -m pip install .
or as "editable":
python -m pip install -e .
Usage
import dask_hpcconfig
To list the available cluster definitions:
dask_hpcconfig.print_clusters()
or, as a mapping of name to type:
clusters = dask_hpcconfig.available_clusters()
To create a cluster, use:
cluster = dask_hpcconfig.cluster(name)
where name
is the name of one of the available clusters.
To override any particular setting: For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:
overrides = {"cluster.n_workers": 7}
cluster = dask_hpcconfig.cluster("datarmor-local", **overrides)
For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :
overrides = {"cluster.cores": 7}
cluster = dask_hpcconfig.cluster("datarmor", **overrides)
cluster.scale(49)
cluster
can then be used to create a Client
:
from distributed import Client
client = Client(cluster)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dask-hpcconfig-2022.3.0.tar.gz
(11.2 kB
view hashes)
Built Distribution
Close
Hashes for dask_hpcconfig-2022.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e80e123b364c5aad5c55abb5ea4a3f504525bb0b7f436df84be4a62d28c6d416 |
|
MD5 | feb2faeb26742380d479372a06aacd95 |
|
BLAKE2b-256 | b98adde3bdf98ab948b8614e155cbe491ddb87b69675538b1ebe2fc0f94bf2dd |