multiprocessing-wrap 0.0.2

Elegant multiprocessing without the boilerplate and confusing syntax

multiprocessing-wrap

A simple interface for writing concurrent scripts. Get the most out of multiprocessing without all the boilerplate and confusing syntax!

Features

• Sensible error propagation - having a stack trace showing where your code speeds debugging and development
• Built-in loading bar as default using tqdm
• Uses dill for pickling, which extends the types that can be passed to your workers (see here for documentation of the limitations of python's default pickling)

Installation

To install using pip:

pip install multiprocessing-wrap

Usage

You can use the functional multiprocess for one line multiprocessing:

from multiprocess import multiprocess

f = lambda: print(1)
multiprocess(f, [(), (), ()])

1  
1  
1

Otherwise you can use the Multiprocess class to use the more explicit add_tasks and do_tasks directives.:

from multiprocess import Multiprocess

m = Multiprocess(show_loading_bar=False)
f = lambda: print(1)
m.add_tasks(f, [(), (), ()])
m.do_tasks() # blocking
m.close()

A more involved example of sorting numbers using sleep. Note that you only have as many workers as you have threads, so if you have 4 threads you will only be able to sort up to 4 numbers with this approach:

from multiprocess import multiprocess, Queue
from time import sleep

def sleep_sort():
  q = Queue()
  def f(q, x):
    sleep(x)
    q.push(x)

  multiprocess(f, [(q, 1,), (q, 2,), (q, 1.5,)])
  print('SORTED')
  while not q.empty():
    print(q.pop())

sleep_sort()

1  
1.5  
2

Error handling

Errors from within a process are propagated back to the parent with stack information. For example:

from multiprocess import Multiprocess

m = Multiprocess()
def f(x):
  raise ValueError('bad error')

m.add_tasks(f, [(1,)])
m.do_tasks()
m.close()

Traceback (most recent call last):
  File "./example3.py", line 9, in <module>
    m.do_tasks()
  File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 53, in do_tasks
    self._check_for_exceptions()
  File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 71, in _check_for_exceptions
    "\n".join(['ERROR: ' + str(e) for e in exceptions]))
multiprocess.MultiprocessException: 1 errors occurred:
ERROR: Error in function call "f((1,))"
Traceback (most recent call last):
  File "/Users/dom/Documents/git/Multiprocess/src/multiprocess.py", line 85, in my_worker
    fn(*rem_args)
  File "./example3.py", line 6, in f
    raise ValueError('bad error')
ValueError: bad error