Skip to main content

EasyCluster: a remote execution/clustering module for Python

Project description


EasyCluster is a remote execution / clustering module for Python.

Possible uses include:
* computation (e.g. NumPy, PyOpenCL)
* coordinated automation for testing networks / SANs
* access to specific hardware in multiple systems (e.g. GPUs, video capture/encoding boards)

* CPython 2.6+ or 3.2+

* Transparent calling of functions and methods
* Transparent handling of exceptions
* Convenience functions for calling one function in parallel on
multiple remote systems
* Automatic support for threading
* Requests and responses protected with shared HMAC key
* Cross-platform compatible; Master scripts running on Linux/OSX can connect
to servers running on Windows and vice/versa.

How it works

EasyCluster works by having a single master script connect to one or more
servers running the cluster service. The master can then call Python functions
on the remote service or send code to execute:

>>> from easycluster import *
>>> define_common('''
... def addvals(a, b):
... return a + b
... def subvals(a, b):
... return a - b
... ''')
>>> key = read_key_file('secret.key')
>>> rmt = Client(key, 'localhost')
>>> rmt.addvals(3, 4)
>>> rmt.subvals(15, 4)
>>>['/bin/echo', 'hello'])

Any "complex" objects that the server returns are converted into proxy objects
on the master which maintain a reference to the object on the server. Calling a
method on this proxy calls the corresponding method on the server. These proxy
objects can also be passed as arguments to other functions on the same
connection, and will be unserialized as the original object on the
server. "Simple" objects like strings, numbers, lists, tuples, etc. are copied
by value.

>>> define_common('''
... class TestObject1(ServerObject):
... def __init__(self, val):
... self.val = val
... def getval(self):
... return self.val
... def newobj(self):
... return TestObject1(self.val + 1)
... ''')
>>> rmt.update_definitions()
>>> obj1 = rmt.TestObject1(100)
>>> obj1
<RemoteProxy for oid 1 on localhost:11999>
>>> obj1.getval()
>>> obj2 = obj1.newobj()
>>> obj2
<RemoteProxy for oid 2 on localhost:11999>
>>> obj2.getval()

Classes can indicate that they should be proxied rather than copied by
inheriting from ServerObject. Existing classes which are unaware of EasyCluster
can be registered on the server by calling make_server_class.

There are three ways classes can specify which methods and attributes to export:
* Using the default proxy
Classes which inherit from ServerObject but do not specify a proxy class or
exported methods and attributes will use a "default" proxy which treats all
attributes as methods. This is the method shown above.
* Specifying export_attrs and export_methods
Classes which specify export_methods or export_attrs will have an anonymous
proxy class created on the client which contains wrappers for only the
specified methods and attributes:

>>> define_common('''
... class TestObject2(TestObject1):
... export_methods = ('getval',)
... export_attrs = ('val',)
... ''')
>>> rmt.update_definitions()
>>> obj2 = rmt.TestObject2(200)
>>> obj2.val
>>> obj2.getval()
>>> obj2.non_existant_method()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'dynamic_proxy_getval_val' object has no attribute 'non_existant_method'

* Defining a proxy class directly
This is the most flexible way of exporting methods and attributes. This allows
you to not only define proxy methods and attributes, but allows you to:
** Subclass a proxy class for another object which your object inherits from
without worrying about updating your export_methods and export_attrs when the
base class changes.
** Implement simple methods on the client. For example, most iterators simply
return 'self' from __iter__. In fact, easycluster provides a proxy class you
can inherit from called SelfIterProxy which does this.
** Implement caching of attributes which you know will not change.

>>> define_common('''
... class TestObject3Proxy(RemoteProxy):
... proxy_methods = ('getval',)
... proxy_attrs = ('val',)
... class TestObject3(TestObject1):
... proxy_class = TestObject3Proxy
... ''')
>>> rmt.update_definitions()
>>> obj3 = rmt.TestObject3(300)
>>> type(obj3)
<class 'easycluster_code.TestObject3Proxy'>
>>> obj3.val

The master script can connect to the same server multiple times. Each connection
creates a separate process with a clean environment. The master can also create
a "local" instance, which starts a new server process without having to run a
separate server.

See for an example of how to use most of the features.

EasyCluster is intended to run on a local area network to coordinate specific
tasks. It is not intended as a generic "cloud computing" service. Although data
is signed with a private HMAC key, it is not encrypted, so it should not be used
openly on the Internet with sensitive data. If you want to use EasyCluster to
coordinate systems in remote geographic areas, consider using a VPN or SSH
tunnel. The EasyCluster service operates over a single TCP port, so most
tunneling solutions will work.


EasyCluster uses setuptools for installation. To install, run:

python build
sudo python install

If you do not have setuptools installed, it will be downloaded for you.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
EasyCluster-0.04-py2.7.egg (48.1 kB) Copy SHA256 hash SHA256 Egg 2.7
EasyCluster-0.04.tar.gz (29.0 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page