unidist 0.1.0

Unified Distributed Execution

Unified Distributed Execution

What is unidist?

unidist is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:

• Ray
• MPI
• Dask Distributed
• Python Multiprocessing

unidist is designed to work in a task-based parallel model.

Also, the framework provides a sequential Python backend, that can be used for debugging.

Installation

Currently, unidist can be installed from sources using pip:

# Dependencies for `MultiProcessing` and `Python` backends will be installed as these are supported by default
$ pip install git+https://github.com/modin-project/unidist
# Dependencies for all the execution backends will be installed
$ pip install git+https://github.com/modin-project/unidist#egg=unidist[all]
# Dependencies for `Ray` execution backend will be installed
$ pip install git+https://github.com/modin-project/unidist#egg=unidist[ray]

This will install unidist directly from the repo without you having to manually clone it! Please be aware that the latest changes have not made it into a release and may not be completely stable.

Choosing an execution backend

There are several ways to choose the execution backend for distributed computation. First, the recommended way is to use unidist CLI options:

# Running the script with unidist on Ray backend
$ unidist script.py --backend ray
# Running the script with unidist on Dask backend
$ unidist script.py --backend dask

Second, setting the environment variable:

# unidist will use Ray backend to distribute computations
export UNIDIST_BACKEND=ray
# unidist will use Dask backend to distribute computations
export UNIDIST_BACKEND=dask

Third, using config API directly in your script:

import unidist.config as cfg
cfg.Backend.put("ray") # unidist will use Ray backend to distribute computations
import unidist.config as cfg
cfg.Backend.put("dask") # unidist will use Dask backend to distribute computations

Default execution backend for unidist is Ray.

Usage

unidist provides CLI interface to run python programs.

import unidist
unidist.init() # Ray backend is used by default

@unidist.remote
def foo(x):
    return x * x

# This will run `foo` on a pool of workers in parallel;
# `refs` will contain object references to actual data
refs = [foo.remote(i) for i in range(5)]
# To get the data call `unidist.get(...)`
print(unidist.get(refs))
[0, 1, 4, 9, 16]

For more examples refer to Getting Started section in our documentation.

Full Documentation

Visit the complete documentation on readthedocs: https://unidist.readthedocs.io.