Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Afar

conda-forge pypi PyPI - Python Version License Tests Coverage

One man's magic is another man's engineering
Robert A. Heinlein


Installation

afar may be installed with pip:

pip install afar

or with conda:

conda install -c conda-forge afar

What is it?

afar allows you to run code on a remote Dask cluster using context managers and IPython magics. For example:

import afar
from dask.distributed import Client
client = Client()

with afar.run, remotely:
    import dask_cudf
    df = dask_cudf.read_parquet("s3://...")
    result = df.sum().compute()

Outside the context, result is a Dask Future whose data resides on a worker. result.result() is necessary to copy the data locally.

Variables are set outside the context at module scope and in IPython or Jupyter. Inside a function, they are set on Python 3.13 and later; on earlier versions, use the dictionary afar.run returns (with afar.run as data, remotely:), described below.

By default, only the last assignment is saved. One can specify which variables to save:

with afar.run("one", "two"), remotely:
    one = 1
    two = one + 1

one and two are now both Futures. They can be used directly in other afar.run contexts:

with afar.run as data, remotely:
    three = one + two

assert three.result() == 3
assert data["three"].result() == 3

data above is a dictionary of variable names to Futures. It may be necessary at times to get the data from here. Alternatively, you may pass a mapping to afar.run to use as the data.

run = afar.run(data={"four": 4})
with run, remotely:
    seven = three + four
assert run.data["seven"].result() == 7

If you want to automatically gather the data locally (to avoid calling .result()), use afar.get instead of afar.run:

with afar.get, remotely:
    five = two + three
assert five == 5

Interactivity in Jupyter

There are several enhancements when using afar in Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend.

The rich repr of the final expression will be displayed if it's not an assignment:

with afar.run, remotely:
    three + seven
# displays 10!

As in IPython, a final expression that ends with ;, or whose value is None, is not displayed.

Printing is captured and displayed locally and asynchronously:

with afar.run, remotely:
    print(three)
    print(seven, file=sys.stderr)
# 3
# 7

These are done asynchronously using ipywidgets. Errors raised remotely are shown the same way, as tracebacks.

Magic!

First load afar magic extension:

%load_ext afar

Now you can use afar as line or cell magic. %%afar is like with afar.run, remotely:. It can optionally accept a list of variable names to save:

%%afar x, y
x = 1
y = x + 1

and

z = %afar x + y

z is a Future of the result (or the result itself with %afar -g, which uses afar.get).

Custom Magic

You can create your own IPython magic like %afar that has arguments baked in:

afar.new_magic("on_gpus", where=remotely(resources={"GPU": 1}))

then:

%%on_gpus
import dask_cudf
df = dask_cudf.read_parquet("s3://...")
result = df.sum().compute()

Is this a good idea?

I don't know, but it sure is a joy to use 😃 !

For motivation, see https://github.com/dask/distributed/issues/4003

It's natural to be skeptical of unconventional syntax. And magic. afar is both unconventional and magical, yet it also works well and is surprisingly fun! Why not give it a try to see what you think?

We're still exploring the usability of afar and want to hear what you think. As you're learning afar, please ask yourself questions such as:

  • can we spell anything better?
  • does this offer opportunities?
  • what is surprising?
  • what is lacking?

Here's an example of an opportunity:

on_gpus = afar.remotely(resources={"GPU": 1})

with afar.run, on_gpus:
    ...

This now works! Keyword arguments to remotely will be passed to client.submit.

I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)

Caveats and Gotchas

Repeatedly copying data

afar automatically gets the data it needs--and only the data it needs--from the outer scope and sends it to the Dask cluster to compute on. Since we don't know whether local data has been modified between calls to afar, we serialize and send local variables every time we use run or get. This is generally fine: it works, it's safe, and is usually fast enough. However, if you do this frequently with large-ish data, the performance could suffer, and you may be using more memory on your local machine than necessary.

With Dask, a common pattern is to send data to the cluster with scatter and get a Future back. This works:

A = np.arange(10**7)
A = client.scatter(A)
with afar.run, remotely:
    B = A + 1
# A and B are now both Futures; their data is on the cluster

Another option is to pass data to run:

run = afar.run(data={"A": np.arange(10**7)})
with run, remotely:
    B = A + 1
# run.data["A"] and B are now both Futures; their data is on the cluster

Here's a nifty trick to use if you're in an IPython notebook: use data=globals()!

run = afar.run(data=globals())
A = np.arange(10**7)
with run, remotely:
    B = A + 1
# A and B are now both Futures; their data is on the cluster

Mutating remote data

As with any Dask workload, one should be careful to not modify remote data that may be reused.

Mutating local data

Similarly, code run remotely isn't able to mutate local variables. For example:

d = {}
with afar.run, remotely:
    d['key'] = 'value'
# d == {}

✨ This code is highly experimental and magical! ✨

Metadata

Release files for afar 0.7.0a0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for afar 0.7.0a0
File Size Uploaded Provenance Actions
afar-0.7.0a0.tar.gz 39.5 kB With provenance Details for afar-0.7.0a0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for afar 0.7.0a0
File Interpreter ABI Platform Actions
afar-0.7.0a0-py3-none-any.whl Python 3 none any Details for afar-0.7.0a0-py3-none-any.whl

Total release size: 78.8 kB

Release files / afar-0.7.0a0.tar.gz

Download URL afar-0.7.0a0.tar.gz
Size 39.5 kB
Tags Source
SHA-256 checksum
How to use checksums
beb317813b173943a865d1dfd86f3109a335b362e01cb6323a05a875ab52bb6c
BLAKE2b-256 checksum
How to use checksums
4927cb534128aa001fe8fd35028577818187b9f2a1e125afaf534f4fb9bab5bc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / afar-0.7.0a0-py3-none-any.whl

Download URL afar-0.7.0a0-py3-none-any.whl
Size 39.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2a3de82fad1c0822bab3e7b28850bd0bd02a59027fa574b361718b2249577114
BLAKE2b-256 checksum
How to use checksums
3be538533f2ee8b316ffab55fe868dcadd4b2ebef016e8a23713ec6c20249cb0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Security for afar 0.7.0a0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for afar-0.7.0a0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page