Skip to main content

Elegant multiprocessing without the boilerplate and confusing syntax

Project description


Build Status PyPI shield

A simple interface for writing concurrent scripts. Get the most out of multiprocessing without all the boilerplate and confusing syntax!


  • Sensible error propagation - having a stack trace showing where your code speeds debugging and development
  • Built-in loading bar as default using tqdm
  • Uses dill for pickling, which extends the types that can be passed to your workers (see here for documentation of the limitations of python's default pickling)


To install using pip:

pip install multiprocessing-wrap


You can use the functional multiprocess for single line multiprocessing:

from multiprocess import multiprocess

f = lambda: print(1)
multiprocess(f, [(), (), ()])

Otherwise you can use the Multiprocess class to use the more explicit add_tasks and do_tasks directives:

from multiprocess import Multiprocess

m = Multiprocess(show_loading_bar=False)
f = lambda: print(1)
m.add_tasks(f, [(), (), ()])
m.do_tasks() # blocking

A more involved example of sorting numbers using sleep. Since the worker function is run in a different process, to transfer data between the processes we use a thread-safe Queue. Note the following only works if you have at least 2 threads:

from multiprocess import multiprocess, Queue
from time import sleep

def sleep_sort():
  q = Queue()
  def f(q, x):

  multiprocess(f, [(q, 1,), (q, 2,)])
  while not q.empty():


Error handling

Errors from within a process are propagated back to the parent with stack information. For example:

from multiprocess import Multiprocess

m = Multiprocess()
def f(x):
  raise ValueError('bad error')

m.add_tasks(f, [(1,)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dom/Documents/git/Multiprocess/multiprocess/", line 61, in do_tasks
  File "/Users/dom/Documents/git/Multiprocess/multiprocess/", line 79, in _check_for_exceptions
    "\n".join(['ERROR: ' + str(e) for e in exceptions]))
multiprocess.multiprocessClass.MultiprocessProcessException: 1 errors occurred:
ERROR: Error in function call "f((1,))"
Traceback (most recent call last):
  File "/Users/dom/Documents/git/Multiprocess/multiprocess/", line 95, in my_worker
  File "<stdin>", line 2, in f
ValueError: bad errorr

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for multiprocessing-wrap, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size multiprocessing_wrap-0.0.3-py3-none-any.whl (4.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size multiprocessing_wrap-0.0.3.tar.gz (4.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page