This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

aioprocessing provides non-blocking, asyncio compatible, coroutine versions of many blocking instance methods on objects in the multiprocessing library. Here’s an example showing non-blocking usage of Event, Queue, and Lock:

import time
import asyncio
import aioprocessing
import multiprocessing

def func(queue, event, lock, items):
    """ Demo worker function.

    This worker function runs in its own process, and uses
    normal blocking calls to aioprocessing objects.

    with lock:
        for item in items:

def example(queue, event, lock):
    l = [1,2,3,4,5]
    p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l))
    while True:
        result = yield from queue.coro_get()  # Non-blocking
        if result is None:
        print("Got result {}".format(result))
    yield from p.coro_join()  # Non-blocking

def example2(queue, event, lock):
    yield from event.coro_wait()  # Non-blocking
    with (yield from lock):  # Non-blocking
        yield from queue.coro_put(78)
        yield from queue.coro_put(None) # Shut it down

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    queue = aioprocessing.AioQueue()
    lock = aioprocessing.AioLock()
    event = aioprocessing.AioEvent()
    tasks = [
        asyncio.async(example(queue, event, lock)),
        asyncio.async(example2(queue, event, lock)),

How does it work?

In most cases, this library makes blocking calls to multiprocessing methods non-blocking by executing the call in a ThreadPoolExecutor, using asyncio.run_in_executor(). It does not re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use aioprocessing objects instead of multiprocessing objects, because each one is generally introducing at least one threading.Thread object, along with a ThreadPoolExecutor. It also means that all the normal risks you get when you mix threads with fork apply here, too.

The one exception to this is aioprocessing.AioPool, which makes use of the existing callback and error_callback keyword arguments in the various Pool.*_async methods to run them as asyncio coroutines. Note that multiprocessing.Pool is actually using threads internally, so the thread/fork mixing caveat still applies.

Each multiprocessing class is replaced by an equivalent aioprocessing class, distinguished by the Aio prefix. So, Pool becomes AioPool, etc. All methods that could block on I/O also have a coroutine version that can be used with asyncio. For example, multiprocessing.Lock.acquire() can be replaced with aioprocessing.AioLock.coro_acquire().

What parts of multiprocessing are supported?

Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in aioprocessing that extend the multiprocessing versions by adding coroutine versions of all the blocking methods.

  • Pool
  • Process
  • Lock
  • RLock
  • Semaphore
  • BoundedSemaphore
  • Event
  • Condition
  • Barrier
  • connection.Connection
  • connection.Listener
  • connection.Client
  • Queue
  • JoinableQueue
  • SimpleQueue
  • All managers.SyncManager Proxy versions of the items above (SyncManager.Queue, SyncManager.Lock(), etc.).


This project is currently in alpha stages, and likely has bugs. Use at your own risk. (I do appreciate bug reports, though :).

Release History

Release History


This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
aioprocessing-0.0.1.tar.gz (8.5 kB) Copy SHA256 Checksum SHA256 Source Sep 20, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting