distrunner 0.0.40

This library allows for the easy construction and management of Dask clusters from a Git repository via a simple context manager.

Distributed Runner

This library allows for the easy construction and management of Dask clusters from a Git repository via a simple context manager.

Installation

pip install distrunner

Usage

In your scheduler (Airfow etc.) use this:

from distrunner import DistRunner

with DistRunner(
    workers=5,
    python_version="3.10",
    repo="https://gitlab.com/crossref/labs/task-test.git",
    entry_module="task",
    entry_point="entry_point",
    requirements_file="requirements.txt",
    local=False,
    retries=3,
    worker_memory=16384,
    worker_cpus=4096,
) as dr:
    logging.basicConfig(level=logging.INFO)

    dr.run()

The "local" flag will determine whether a remote cluster is created.

The code in the git repository at the module and entry point that you specify will be called, passing the DaskRunner object. You can use this, then, to obtain a Dask client by calling cldr.client.

You will need to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to use the Fargate clusters.

Features

• Context manager handling of Dask Fargate clusters with scale-to-zero on complete
• Easy ability to switch between local and distributed/remote development
• Simple deployment from a git repository including all requirements
• Bugfixes to Dask AWS 2022.10.0 to suppress errors in weakref finalizers

What it Does

This library allows you to bootstrap a git repository into a distributed computation environment. It will install all the needed dependencies into the current virtual environment and sync these with workers. Your code's entrypoint will be called with access to a Dask Client object.

Credits

• AWS/Boto
• Dask
• Git
• GitPython

Copyright © Crossref 2023