Skip to main content

submitting cpu-bound tasks to processes and io-bound tasks to threads

Project description

Write a classic sequential program. Then convert it into a parallel one.

Why?

It runs faster.

What if not?

Don’t use it.

How?

Before:

for image in images:
    create_thumbnail(image)

After:

from fork import fork

for image in images:
    fork(create_thumbnail, image)

What about return values?

As usual:

result = fork(my_func, *args, **kwargs)

It’s a proxy object that behaves almost exactly like the real return value of my_func except that it’s lazy.

How lazy?

Very lazy. You can even add, multiply, etc. such proxy results without blocking which come in quite handy, especially in loops. Use fork.await, str, print, etc. to force evaluation and get the real and non-lazy value back.

sizes = 0
for image in images:
    sizes += fork(create_thumbnail, image) # lazy evaluation
print(sizes)                               # forces evaluation

Threads or Processes?

You don’t need to bother. fork will take care of that for you.

You can assist fork by decorating your functions; not decorating defaults to fork.cpu_bound:

@io_bound
def call_remote_webservice():
    # implementation

@cpu_bound
def heavy_computation(n):
    # implementation

Exception handling

Original (sequential) tracebacks are preserved. That should make debugging easier. However, don’t try to catch exceptions. You better want to exit and see them. When you force evaluation potential exceptions will be raised.

Advanced Feature: Force Specific Type of Execution

If you really need more control over the type of execution, use fork.process or fork.thread. They work just like fork.fork but enforce the corresponding type of background execution.

import pkg_resources

for worker_function in pkg_resources.iter_entry_points(group='worker'):
    process(worker_function)

Advanced Feature: Multiple Execution At Once

You can shorten your programs by using fork.map. It works like fork.fork but submits a function multiple times for each item given by an iterable.

results = fork.map(create_thumbnail, images)

fork.map_process and fork.map_thread work accordingly and force a specific type of execution. Use those if really necessary. Otherwise, just use fork.map. fork take care for you in this case again.

In order to wait for the completion of a set of result proxies, use fork.await_all. If you want to unblock by the first unblocking result proxy, call fork.await_any.

There are also blocking variants available: fork.block_map, fork.block_map_process and fork.block_map_thread; in case you need some syntactic sugar:

fork.await_all(fork.map(create_thumbnail, images))
# equals
fork.block_map(create_thumbnail, images)

Conclusion

Good

  • easy to give it a try / easy way from sequential to parallel and back

  • results evaluate lazily

  • sequential tracebacks are preserved

  • it’s thread-safe / cascading forks possible

  • compatible with Python 2 and 3

Bad

  • weird calling syntax (no syntax support)

  • type(result) == ResultProxy

  • not working with lambdas due to PickleError

  • needs fix:

    • not working with coroutines (asyncio) yet (working on it)

  • cannot fix efficiently:

    • exception handling (force evaluation when entering and leaving try blocks)

  • ideas are welcome :-)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xfork-0.37.tar.gz (6.4 kB view details)

Uploaded Source

Built Distributions

xfork-0.37-py3-none-any.whl (5.2 kB view details)

Uploaded Python 3

xfork-0.37-py2-none-any.whl (5.2 kB view details)

Uploaded Python 2

File details

Details for the file xfork-0.37.tar.gz.

File metadata

  • Download URL: xfork-0.37.tar.gz
  • Upload date:
  • Size: 6.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for xfork-0.37.tar.gz
Algorithm Hash digest
SHA256 9ce229ae61ddde063d93907897be8442dd31aaf5f0dcec6269089577af2b307f
MD5 8686561c1b0a10e4e40f5554587ecbf9
BLAKE2b-256 6f2ea91a0d5bd83fd48696b729302f6afe9be5420a34829fb456760ac6ec63e6

See more details on using hashes here.

File details

Details for the file xfork-0.37-py3-none-any.whl.

File metadata

File hashes

Hashes for xfork-0.37-py3-none-any.whl
Algorithm Hash digest
SHA256 bec5f12a11746a2ced266348a072d367a159d03056d0dc8d39fa0211b81f8134
MD5 0d5208101bde96ffe32ff0cf69cd9423
BLAKE2b-256 8e1515aa858489048267da37dfc6c3af67ac75ce32bd63f913b270b15db5bf43

See more details on using hashes here.

File details

Details for the file xfork-0.37-py2-none-any.whl.

File metadata

File hashes

Hashes for xfork-0.37-py2-none-any.whl
Algorithm Hash digest
SHA256 1765040264552879c75ea501e3bada2f57627dc1497d4d7579a9ede4c69543a4
MD5 6ae2a5ae7f84150efd087e635e67de57
BLAKE2b-256 32989c137e49d3fa9fb3cb7d96cc89ac4aa63846d0907309ad9e0843aab692cc

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page