Skip to main content

Dask Cluster objects in Saturn Cloud

Project description

dask-saturn

Python library for interacting with Dask clusters in Saturn Cloud.

Dask-Saturn mimics the API of Dask-Kubernetes, but allows the user to interact with clusters created within Saturn Cloud.

Start cluster

From within a Jupyter notebook, you can start a cluster:

from dask_saturn import SaturnCluster

cluster = SaturnCluster()
cluster

By default this will start a dask cluster with the same settings that you have already set in the Saturn UI or in a prior notebook.

To start the cluster with a certain number of workers using the n_workers option. Similarly, you can set the scheduler_size, worker_size, and worker_is_spot.

Note: If the cluster is already running then you can't change the settings. Attempting to do so will raise a warning.

Adjust number of workers

Once you have a cluster you can interact with it via the jupyter widget, or using the scale and adapt methods.

For example, to manually scale up to 20 workers:

cluster.scale(20)

To create an adaptive cluster that controls its own scaling:

cluster.adapt(minimum=1, maximum=20)

Interact with client

To submit tasks to the cluster, you sometimes need access to the Client object. Instantiate this with the cluster as the only argument:

from distributed import Client

client = Client(cluster)
client

Close cluster

To terminate all resources associated with a cluster, use the close method:

cluster.close()

Change settings

To update the settings (such as n_workers, worker_size, worker_is_spot, nthreads) on an existing cluster, use the reset method:

cluster.reset(n_workers=3)

You can also call this without instantiating the cluster first:

cluster = SaturnCluster.reset(n_workers=3)

Development

Create/update a dask-saturn conda environment:

make conda-update

Set environment variables to run dask-saturn with a local atlas server:

export BASE_URL=http://dev.localtest.me:8888/
export SATURN_TOKEN=<JUPYTER_SERVER_SATURN_TOKEN>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-saturn-0.1.3.tar.gz (24.6 kB view hashes)

Uploaded Source

Built Distribution

dask_saturn-0.1.3-py3-none-any.whl (9.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page