Skip to main content

submitting cpu-bound tasks to processes and io-bound tasks to threads

Project description

Convert a classic sequential program into a parallel one.


It runs faster.

What if not?

Don’t use it.


for image in images:
    create_thumbnail(image)       # original

for image in images:
    fork(create_thumbnail, image) # parallelized explicitly

for image in images:
    create_thumbnail(image)       # parallelized implicitly (read below)

What about return values?

result = fork(my_func, *args, **kwargs)

And what is this result?

A result proxy that behaves almost exactly as if it were the return value of my_func. That in turn means, as soon as you access the result and it is not ready yet, the main thread blocks.

Furthermore, result proxies are contagious, i.e. evaluation are delayed until they are really needed. Don’t panic; in case of an error, you will receive the same traceback that you would see in the sequential case.

Speaking of threads …

and processes? fork will take care of that for you.

You can assist fork by decorating your functions (not decorating defaults to cpu_bound):

def call_remote_webservice():
    # implementation

def fib(n):
    # naive implementation of Fibonacci numbers

@unsafe # don't fork; run sequentially
def weird_side_effects(*args, **kwargs):
    # implementation

Parallelize implicitly?

If you don’t like the fork calling syntax, you can convert certain functions into forks.

Use with caution.

def create_thumbnail_by_webservice(image):
    # implementation

def create_thumbnail_by_bare_processing_power(image):
    # implementation

# the following two lines spawn two forks



  • easy way back and forth (from sequential to parallel and vice versa)
  • cascading possible (thread-safe)
  • compatible with Python 2 and 3


  • weird calling syntax (no syntax support)
  • type(result) == FutureWrapper
  • not working with lambdas due to PickleError
  • needs fix:
    • contagious_result (caller has no control over result; might get deprecated)
    • still needs some mechanism to wait and to evaluate all contagious results (using ‘with’ or function call)
    • not working with coroutines (asyncio) yet

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for xfork, version 0.17
Filename, size File type Python version Upload date Hashes
Filename, size xfork-0.17-py2-none-any.whl (4.1 kB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size xfork-0.17-py3-none-any.whl (4.1 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size xfork-0.17.tar.gz (4.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page