Run code on a dask worker via a context manager
Project description
Afar
One man's magic is another man's engineering
Robert A. Heinlein
afar
allows you to run code on a remote Dask worker using context managers. For example:
import afar
with afar.run, remotely:
import dask_cudf
df = dask_cudf.read_parquet("s3://...")
result = df.sum().compute()
Outside the context, result
is a Dask Future whose data resides on a worker. result.result()
is necessary to copy the data locally.
By default, only the last assignment is saved. One can specify which variables to save:
with afar.run("a", "b"), remotely:
a = 1
b = a + 1
a
and b
are now both Futures. They can be used directly in other afar.run
contexts:
with afar.run as data, remotely:
c = a + b
assert c.result() == 3
assert data["c"].result() == 3
data
is a dictionary of variable names to Futures. It may be necessary at times to get the data from here.
For motivation, see https://github.com/dask/distributed/issues/4003
This code is highly experimental and magical!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.