Skip to main content

submitting cpu-bound tasks to processes and io-bound tasks to threads

Project description

Convert a classic sequential program into a parallel one.

Why?

It runs faster.

What if not?

Don’t use it.

How?

for image in images:
    create_thumbnail(image)       # original

for image in images:
    fork(create_thumbnail, image) # parallelized explicitly

for image in images:
    create_thumbnail(image)       # parallelized implicitly (read below)

Does it work with return values?

Sure:

result = fork(my_func, *args, **kwargs)

It’s a proxy object that behaves almost exactly like the real return value of my_func.

Furthermore, it evaluates only if needed; also in combination with operators (like +, - etc.).

What about exceptions?

Its original (sequential) traceback is preserved. That should make debugging easier.

Speaking of threads …

and processes? fork will take care of that for you.

You can assist fork by decorating your functions (not decorating defaults to cpu_bound):

@io_bound
def call_remote_webservice():
    # implementation

@cpu_bound
def fib(n):
    # naive implementation of Fibonacci numbers

@unsafe # don't fork; run sequentially
def weird_side_effects(*args, **kwargs):
    # implementation

Parallelize implicitly?

If you don’t like the fork calling syntax, you can convert certain functions into forks.

Use with caution.

@io_bound_fork
def create_thumbnail_by_webservice(image):
    # implementation

@cpu_bound_fork
def create_thumbnail_by_bare_processing_power(image):
    # implementation

# the following two lines spawn two forks
create_thumbnail_by_webservice(image1)
create_thumbnail_by_bare_processing_power(image2)

Conclusion

Good

  • easy way back and forth (from sequential to parallel and vice versa)

  • results evaluate lazily

  • tracebacks are preserved

  • cascading possible (thread-safe)

  • compatible with Python 2 and 3

Bad

  • weird calling syntax (no syntax support)

  • type(result) == ResultProxy

  • not working with lambdas due to PickleError

  • needs fix:

    • “maximum recursion depth exceeded” due to encapsulating all operations into proxies

    • not working with coroutines (asyncio) yet

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xfork-0.22.tar.gz (5.0 kB view hashes)

Uploaded Source

Built Distributions

xfork-0.22-py3-none-any.whl (4.4 kB view hashes)

Uploaded Python 3

xfork-0.22-py2-none-any.whl (4.4 kB view hashes)

Uploaded Python 2

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page