xfork 0.16

submitting cpu-bound tasks to processes and io-bound tasks to threads

Convert a classic sequential program into a parallel one.

Why?

It runs faster.

What if not?

Don’t use it.

How?

for image in images:
    create_thumbnail(image)       # original

for image in images:
    fork(create_thumbnail, image) # parallelized explicitly

for image in images:
    create_thumbnail(image)       # parallelized implicitly (read below)

What about return values?

result = fork(my_func, *args, **kwargs)

And what is this result?

A future that behaves almost exactly as if it were the return value of my_func. That in turn means, as soon as you access the result and it is not ready yet, the main thread blocks.

Speaking of threads …

and processes? fork will take care of that for you.

You can assist fork by decorating your functions (not decorating defaults to cpu_bound):

@io_bound
def call_remote_webservice():
    # implementation

@cpu_bound
def fib(n):
    # naive implementation of Fibonacci numbers

@unsafe # don't fork; run sequentially
def weird_side_effects(*args, **kwargs):
    # implementation

Parallelize implicitly?

If you don’t like the fork calling syntax, you can convert certain functions into forks.

Use with caution.

@io_bound_fork
def create_thumbnail_by_webservice(image):
    # implementation

@cpu_bound_fork
def create_thumbnail_by_bare_processing_power(image):
    # implementation

# the following two lines spawn two forks
create_thumbnail_by_webservice(image1)
create_thumbnail_by_bare_processing_power(image2)

I still need more performance.

You feel like debugging is still too easy, don’t you? Go ahead with contagious futures.

Use with extreme caution.

NOTE: decorator ‘contagious’ was renamed to ‘contagious_result’.

@io_bound
def item():
    # implementation

result = 0
for item in items:
    result += fork_contagious(item) # explicit
print(result)

# or

@io_bound # also works with fork decorator
@contagious_result
def item():
    # implementation

result = 0
for item in items:
    result += fork(item)            # implicit
print(result)

Conclusion

Good

• easy way back and forth (from sequential to parallel and vice versa)

• cascading possible (thread-safe)

• compatible with Python 2 and 3

Bad

• weird calling syntax (no syntax support)

• type(result) == FutureWrapper

• not working with lambdas due to PickleError

• needs fix:

  • contagious_result (caller has no control over result; might get deprecated)

  • still needs some mechanism to wait and to evaluate all contagious results (using ‘with’ or function call)

  • not working with coroutines (asyncio) yet