dask-jobqueue 0.1.0

Deploy Dask on job queuing systems like PBS or SLURM

This helps to deploy Dask on batch-style job schedulers like PBS and SLURM.

Example

from dask_jobqueue import PBSCluster

cluster = PBSCluster(processes=6, threads=4, memory="16GB")
cluster.start_workers(10)

from dask.distributed import Client
client = Client(cluster)

Adaptivity

This can also adapt the cluster size dynamically based on current load. This helps to scale up the cluster when necessary but scale it down and save resources when not actively computing.

cluster.adapt()

History

This package came out of the Pangeo collaboration and was copy-pasted from a live repository at this commit. Unfortunately, development history was not preserved.

Original developers include the following:

• Jim Edwards

• Joe Hamman

• Matthew Rocklin