distrunner 0.0.1

This library allows for the easy construction and management of Dask clusters from a Git repository via a simple context manager.

Distributed Runner

This library allows for the easy construction and management of Dask clusters from a Git repository via a simple context manager.

Installation

pip install distrunner

Usage

In your scheduler (Airfow etc.) use this:

from distrunner import DistRunner

with DistRunner(
    workers=5,
    python_version="3.10",
    repo="https://gitlab.com/crossref/labs/task-test.git",
    entry_module="task",
    entry_point="entry_point",
    requirements_file="requirements.txt",
    local=False,
    retries=3,
    worker_memory=16384,
    worker_cpus=4096,
) as dr:
    logging.basicConfig(level=logging.INFO)

    dr.run()

The "local" flag will determine whether a remote cluster is created.

The code in the git repository at the module and entry point that you specify will be called, passing the DaskRunner object. You can use this, then, to obtain a Dask client by calling cldr.client.

Features

• Context manager handling of Dask Fargate clusters with scale-to-zero on complete
• Easy ability to switch between local and distributed/remote development
• Simple deployment from a git repository including all requirements
• Bugfixes to Dask AWS 2022.10.0 to suppress errors in weakref finalizers

Credits

• AWS/Boto
• Dask
• Git
• GitPython

Copyright © Crossref 2023