ncar-jobqueue 2020.11.18

Utilities for expanding dask-jobqueue with appropriate settings for NCAR's clusters

ncar-jobqueue

• ncar-jobqueue
  • Badges
  • Installation
  • Usage
    • Casper
    • Cheyenne
    • Hobart
    • Izumi
    • Non-NCAR machines

Badges

CI
Docs
Package
License

Utilities for expanding dask-jobqueue with appropriate settings for NCAR's clusters.

Supported clusters:

• Cheyenne
• Casper (DAV)
• CGD's Hobart
• CGD's Izumi

Installation

NCAR-jobqueue can be installed from PyPI with pip:

python -m pip install ncar-jobqueue

(COMING SOON) NCAR-jobqueue will also be available from conda-forge for conda installations:

conda install -c conda-forge ncar-jobqueue

Usage

Casper

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=2, memory=50.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Cheyenne

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=72, memory=218.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Hobart

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=96, memory=192.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Izumi

>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=96, memory=192.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)

Non-NCAR machines

On non-NCAR machines, ncar-jobqueue will warn the user, and it will use distributed.LocalCluster:

>>> from ncar_jobqueue import NCARCluster
.../ncar_jobqueue/cluster.py:42: UserWarning: Unable to determine which NCAR cluster you are running on... Returning a `distributed.LocalCluster` class.
warn(message)
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster('tcp://127.0.0.1:49334', workers=4, threads=8, memory=17.18 GB)