afar 0.1.0

Run code on a dask worker via a context manager

Afar

One man's magic is another man's engineering
Robert A. Heinlein

---

afar allows you to run code on a remote Dask worker using context managers. For example:

import afar

with afar.run, remotely:
    import dask_cudf
    df = dask_cudf.read_parquet("s3://...")
    result = df.sum().compute()

Outside the context, result is a Dask Future whose data resides on a worker. result.result() is necessary to copy the data locally.

By default, only the last assignment is saved. One can specify which variables to save:

with afar.run("a", "b"), remotely:
    a = 1
    b = a + 1

a and b are now both Futures. They can be used directly in other afar.run contexts:

with afar.run as data, remotely:
    c = a + b

assert c.result() == 3
assert data["c"].result() == 3

data is a dictionary of variable names to Futures. It may be necessary at times to get the data from here.

For motivation, see https://github.com/dask/distributed/issues/4003

This code is highly experimental and magical!