ncar-jobqueue 2020.3.4

Utilities for expanding dask-jobqueue with appropriate settings for NCAR's clusters

ncar-jobqueue

Utilities for expanding dask-jobqueue with appropriate settings for NCAR’s clusters.

Supported clusters:

• Cheyenne

• Casper (DAV)

• CGD’s Hobart

Installation

ncar-jobqueue can be installed from PyPI with pip:

pip install ncar-jobqueue

Usage

Casper

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(2)
>>> cluster
NCARCluster(cores=2, memory=50.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Cheyenne

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(2)
>>> cluster
NCARCluster(cores=72, memory=218.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Hobart

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(2)
>>> cluster
NCARCluster(cores=96, memory=192.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Non-NCAR machines

On non-NCAR machines, ncar-jobqueue will warn the user, and it will use distributed.LocalCluster:

>>> from ncar_jobqueue import NCARCluster
.../ncar_jobqueue/cluster.py:42: UserWarning: Unable to determine which NCAR cluster you are running on... Returning a `distributed.LocalCluster` class.
warn(message)
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster('tcp://127.0.0.1:49334', workers=4, threads=8, memory=17.18 GB)