Skip to main content

submitting cpu-bound tasks to processes and io-bound tasks to threads

Project description

Write a classic sequential program. Then convert it into a parallel one.

Why?

It runs faster.

What if not?

Don’t use it.

How?

for image in images:
    create_thumbnail(image)       # original

for image in images:
    fork(create_thumbnail, image) # parallelized explicitly

for image in images:
    create_thumbnail(image)       # parallelized implicitly (read below)

What about return values?

As usual:

result = fork(my_func, *args, **kwargs)

It’s a proxy object that behaves almost exactly like the real return value of my_func. Furthermore, it evaluates only if needed; also in combination with operators (like +, - etc.).

What happens to exceptions?

Their original (sequential) tracebacks are preserved. That should make debugging easier. However, don’t try to catch exceptions. You better want to exit and see them.

Speaking of threads …

and processes? fork will take care of that for you.

You can assist fork by decorating your functions (not decorating defaults to cpu_bound):

@io_bound
def call_remote_webservice():
    # implementation

@cpu_bound
def fib(n):
    # naive implementation of Fibonacci numbers

@unsafe # don't fork; run sequentially
def weird_side_effects(*args, **kwargs):
    # implementation

Parallelize implicitly?

If you don’t like the fork calling syntax, you can convert certain functions into stand-alone forks.

Use with caution.

@io_bound_fork
def create_thumbnail_by_webservice(image):
    # implementation

@cpu_bound_fork
def create_thumbnail_by_bare_processing_power(image):
    # implementation

# the following two lines spawn two forks
create_thumbnail_by_webservice(image1)
create_thumbnail_by_bare_processing_power(image2)

Conclusion

Good

  • easy to give it a try / easy way from sequential to parallel and back

  • results evaluate lazily

  • sequential tracebacks are preserved

  • it’s thread-safe / cascading forks possible

  • compatible with Python 2 and 3

Bad

  • weird calling syntax (no syntax support)

  • type(result) == ResultProxy

  • not working with lambdas due to PickleError

  • needs fix:

    • fix exception handling somehow

    • not working with coroutines (asyncio) yet

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xfork-0.25.tar.gz (5.1 kB view details)

Uploaded Source

Built Distributions

xfork-0.25-py3-none-any.whl (4.4 kB view details)

Uploaded Python 3

xfork-0.25-py2-none-any.whl (4.4 kB view details)

Uploaded Python 2

File details

Details for the file xfork-0.25.tar.gz.

File metadata

  • Download URL: xfork-0.25.tar.gz
  • Upload date:
  • Size: 5.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for xfork-0.25.tar.gz
Algorithm Hash digest
SHA256 bdc6006bc023bfb8f8a8eb0d15d57788cdc2dec5694e59d587b3d7721dd54527
MD5 1ecb536d47ef36fd67abc564af74ff05
BLAKE2b-256 219bfda6e98e2bcf5f9e0da4ea380cadaa3af625404afb32292c355d77e1e53d

See more details on using hashes here.

File details

Details for the file xfork-0.25-py3-none-any.whl.

File metadata

  • Download URL: xfork-0.25-py3-none-any.whl
  • Upload date:
  • Size: 4.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for xfork-0.25-py3-none-any.whl
Algorithm Hash digest
SHA256 9f8f821997a4fd3fdeddb00ae517aad4b92ffe0e579a80565a1a46691a74d0a6
MD5 e470a5e9766204ffebcd1f88f6701b4b
BLAKE2b-256 f51a10639c3b24370b6dc9031c2747f30d1a708fef95c6b49d519f2485670cb2

See more details on using hashes here.

File details

Details for the file xfork-0.25-py2-none-any.whl.

File metadata

  • Download URL: xfork-0.25-py2-none-any.whl
  • Upload date:
  • Size: 4.4 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for xfork-0.25-py2-none-any.whl
Algorithm Hash digest
SHA256 1961272ab666dc833119a19366b6345bd4b5b522357735e2cd750a050443ba58
MD5 2a0085fee3c64b0d8df02d1bf98f7996
BLAKE2b-256 5b45104f6d7e1dc51e594a5ea1c269cdb6e10ff9ee180a23b4f8fed8311591c9

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page