Skip to main content

Dask on DRMAA

Project description

Build Status PyPI Release conda-forge Release

Deploy a Dask.distributed cluster on top of a cluster running a DRMAA-compliant job scheduler.


Launch from Python

from dask_drmaa import DRMAACluster
cluster = DRMAACluster()

from dask.distributed import Client
client = Client(cluster)

>>> future = client.submit(lambda x: x + 1, 10)
>>> future.result()

Or launch from the command line:

$ dask-drmaa 10  # starts local scheduler and ten remote workers


Python packages are available from PyPI and can be installed with pip:

pip install dask-drmaa

Also conda packages are available from conda-forge:

conda install -c conda-forge dask-drmaa

Additionally the package can be installed from GitHub with the latest changes:

pip install git+ --upgrade


git clone
cd dask-drmaa
pip install .

You must have the DRMAA system library installed and be able to submit jobs from your local machine. Please make sure to set the environment variable DRMAA_LIBRARY_PATH to point to the location of for your system.


This repository contains a Docker-compose testing harness for a Son of Grid Engine cluster with a master and two slaves. You can initialize this system as follows:

docker-compose build

If you have done this previously and need to refresh your solution you can do the following

docker-compose stop
docker-compose build --no-cache

And run tests with py.test in the master docker container

docker exec -it sge_master /bin/bash -c "cd /dask-drmaa; python develop"
docker exec -it sge_master /bin/bash -c "cd /dask-drmaa; py.test dask_drmaa --verbose"

Adaptive Load

Dask-drmaa can adapt to scheduler load, deploying more workers on the grid when it has more work, and cleaning up these workers when they are no longer necessary. This can simplify setup (you can just leave a cluster running) and it can reduce load on the cluster, making IT happy.

To enable this, call the adapt method of a DRMAACluster. You can submit computations to the cluster without ever explicitly creating workers.

from dask_drmaa import DRMAACluster
from dask.distributed import Client

cluster = DRMAACluster()
client = Client(cluster)

futures =, seq)  # workers will be created as necessary


The DRMAA interface is the lowest common denominator among many different job schedulers like SGE, SLURM, LSF, Torque, and others. However, sometimes users need to specify parameters particular to their cluster, such as resource queues, wall times, memory constraints, etc..

DRMAA allows users to pass native specifications either when constructing the cluster or when starting new workers:

cluster = DRMAACluster(template={'nativeSpecification': '-l h_rt=01:00:00'})
# or
cluster.start_workers(10, nativeSpecification='-l h_rt=01:00:00')

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-drmaa-0.2.1.tar.gz (26.0 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page