concurrent processing foundations for Python: events, threads, and processes
Do you concur?
concur is a Work in Progress.
concur is set of modules for working with concurrent tasks. It attempts to provide a consistent programmer interface to scheduling such that tasks can be performed using threads or processes or events without changes to code. Naturally, there is some consistency degradation when using processes as memory boundaries are being crossed, but for common cases involving serializable objects, it can operate transparently.
Concur’s concur.libcontain module provides tools for working with the result of an arbitrary Python task, a callable. It’s a fundamental component for managing completion notifications as it allows you to refer to the exception as well as the return.
Contained returns are straightforward:
from concur import libcontain def returns(): return 5 c = libcontain.Contain(returns) assert c.failed is False assert c.open() == 5
However, contained raises are a little more amorphous:
class LocalException(Exception): pass def raises(): raise LocalException("error") c = libcontain.Contain(raises) assert c.failed is True try: c.open() except libcontain.Containment as err: assert isinstance(err.__cause__, LocalException)
Raises have to raise a new exception every time in order to avoid scribling on the original exception’s tracebacks. Advanced use of containment can raise the original cause, but such use requires care in order to avoid corrupting the traceback stack with frames from other raises.
The concur.libevent module contains implementations for arbitrary joins and event driven queues, a callback deque.
Joins are devices that allow a set of endpoints to be defined such that all endpoints must be called in order for a commit to occur. The commit operation performs the final join, a call to another endpoint that recieves the mapping of endpoint identifiers to containers.
Delivery objects are a trivialization Deferred’s. Concur uses containment to pass error cases, so a more complication construct is not necessary.
The concur.libthread module primarily contains references to thread primitives provided by Python’s _thread module. However, it has an implementation of a deque-based Queue, and a simple Transition primitive for passing objects from thread to thread.
The concur.libprocess module contains primitives for working with isolated processes. Currently, support is limited.