distrunner 1.1.0

This library allows for the easy construction and management of Dask clusters.

Distributed Runner

This library allows for the easy construction and management of Dask clusters via a simple context manager.

Installation

pip install distrunner

Usage

In your scheduler (Airfow etc.) use something like this:

from distrunner import DistRunner
import whatismyip

def main():
    with DistRunner(workers=10) as cldr:
        results = cldr.client.map(print_ip, range(10))
        outcome = cldr.client.gather(results)

        print(outcome)

def print_ip(x):
    return f"My IP address is {whatismyip.whatismyip()}"

The "local" flag will determine whether a remote cluster is created. For example, the following will all run locally instead of spinning up infrastructure:

from distrunner import DistRunner
import whatismyip

def main():
    with DistRunner(workers=10, local=True) as cldr:
        results = cldr.client.map(print_ip, range(10))
        outcome = cldr.client.gather(results)

        print(outcome)

def print_ip(x):
    return f"My IP address is {whatismyip.whatismyip()}"

You will need to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to use the Fargate clusters (or run from an environment with an authorised IAM role).

Features

• Context manager handling of Dask Fargate clusters with scale-to-zero on complete
• Easy ability to switch between local and distributed/remote development

What it Does

This library allows you to easily run functions across a Dask cluster.

Credits

• AWS/Boto
• Coiled
• Dask
• Git
• GitPython

Copyright © Crossref 2023