submitting cpu-bound tasks to processes and io-bound tasks to threads
Project description
Convert a classic sequential program into a parallel one.
Why?
It runs faster.
What if not?
Don’t use it.
How?
for image in images:
create_thumbnail(image) # original
for image in images:
fork(create_thumbnail, image) # parallelized explicitly
for image in images:
create_thumbnail(image) # parallelized implicitly (read below)
What about return values?
result = fork(my_func, *args, **kwargs)
And what is this result?
A future that behaves almost exactly as if it were the return value of my_func. That in turn means, as soon as you access the result and it is not ready yet, the main thread blocks.
Speaking of threads …
and processes? fork will take care of that for you.
You can assist fork by decorating your functions (not decorating defaults to cpu_bound):
@io_bound
def call_remote_webservice():
# implementation
@cpu_bound
def fib(n):
# naive implementation of Fibonacci numbers
@unsafe # don't fork; run sequentially
def weird_side_effects(*args, **kwargs):
# implementation
Parallelize implicitly?
If you don’t like the fork calling syntax, you can convert certain functions into forks.
Use with caution
@io_bound_fork
def create_thumbnail_by_webservice(image):
# implementation
@cpu_bound_fork
def create_thumbnail_by_bare_processing_power(image):
# implementation
# the following two lines spawn two forks
create_thumbnail_by_webservice(image1)
create_thumbnail_by_bare_processing_power(image2)
Contagious futures?
You feel like debugging is too easy and you still need more performance? Go ahead.
Use with extreme caution
@io_bound
def item():
# implementation
result = 0
for item in items:
result += fork_contagious(item) # explicit
print(result)
# or
@io_bound # also works with fork decorator
@contagious
def item():
# implementation
result = 0
for item in items:
result += fork(item) # implicit
print(result)
Conclusion
Good
easy way back and forth (from sequential to parallel and vice versa)
cascading possible (thread-safe)
compatible with Python 2 and 3
Bad
weird calling syntax (no syntax support)
type(result) == BlockingFuture
not working with coroutines (asyncio) yet
not working with lambdas due to PickleError
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.