Skip to main content

Distributed computing made easier, using remote objects

Project description


| Distributed computing made easier, using remote objects
| N.B. this is a development pre-release: lots not yet working!


Distob will take your existing python objects, or a sequence of objects,
and scatter them onto many IPython parallel engines, which may be
running on a single computer or on a cluster.

In place of the original objects, proxy objects are kept on the client
computer that provide the same interface as the original objects. You
can continue to use these as if the objects were still local. All
methods are passed through to the remote objects, where computation is

In particular, sending numpy arrays to the cluster is supported. (Will
require numpy 1.9.0b1 or later for full functionality with remote

A numpy array can also be scattered across the cluster, along a particular
axis. Operations on the array will then automatically be split up and done
in parallel (still work in progress).

Distob is an object layer built on top of IPython.parallel, so it will
make use of your default IPython parallel profile. This allows different
cluster architectures, local CPUs, SSH nodes, PBS, Amazon EC2, etc.


| ``scatter(obj)`` Distribute any object (or list of objects) to remote iPython engines, return a proxy.
| ``gather(obj)`` Fetch back a distributed object (or list), making it local again.

distributed numpy arrays

| ``scatter(a, axis=2)`` Distribute a single numpy array along axis 2, returning a DistArray.
| ``concatenate``, ``vstack``, ``hstack``, ``dstack``, ``expand_dims``, ``transpose``, ``rollaxis``:
| These work like the numpy functions of the same name. But these can be used with a mix of ordinary ndarrays, RemoteArrays and DistArrays, performing array structural changes while keeping the actual data distributed across multiple engines.
| For example, stacking several RemoteArrays gives a DistArray, without needing to move data.
| The distributed arrays so far support basic indexing, slices and advanced integer indexing.


| ``RemoteArray`` proxy object representing a remote numpy ndarray
| ``DistArray`` a single ndarray distributed across multiple engines
| ``Remote`` base class, used when auto-creating ``Remote*`` proxy classes
| ``@proxy_methods(base)`` class decorator for auto-creating ``Remote*`` proxy classes
| ``ObjectHub`` dict interface giving refs to all distributed objects cluster-wide
| ``ObjectEngine`` dict holding the distributed objects of a single IPython engine
| ``Ref`` reference to a (possibly remote) object


``engine``: the ``ObjectEngine`` instance on each host (``ObjectHub`` on
the client)


- Blocking/non-blocking proxy methods

- Assigning to slices of remote arrays

- Finish implementing remote ufunc support for arrays, with computation routed according to operand location.

- Auto-creation of proxy classes at runtime (depends

- Use caching only if specified for a particular method (initially
read-only methods)

- Make proxy classes more robust, adapting ``wrapt``


Incorporates ```` by Jay Hutchinson,

``IPython`` parallel computing, see:

``dill`` by Mike McKerns for object serialization, see:

Project details

Release history Release notifications

History Node


History Node


History Node


History Node


History Node


History Node


This version
History Node


History Node


History Node


History Node


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
distob-0.1.3.tar.gz (42.1 kB) Copy SHA256 hash SHA256 Source None Aug 25, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page