FORK
Write a classic sequential program. Then convert it into a parallel one.
Why?
It runs faster.
What if not?
Don’t use it.
How?
Before:
for image in images:
create_thumbnail(image)After:
from fork import fork
for image in images:
fork(create_thumbnail, image)What about return values?
As usual:
result = fork(my_func, *args, **kwargs)It’s a proxy object that behaves almost exactly like the real return value of my_func except that it’s lazy.
How lazy?
Very lazy. You can even add, multiply, etc. such proxy results without blocking which come in quite handy, especially in loops. Use fork.await, str, print, etc. to force evaluation and get the real and non-lazy value back.
sizes = 0
for image in images:
sizes += fork(create_thumbnail, image) # lazy evaluation
print(sizes) # forces evaluationThreads or Processes?
You don’t need to bother. fork will take care of that for you.
You can assist fork by decorating your functions; not decorating defaults to fork.cpu_bound:
@io_bound
def call_remote_webservice():
# implementation
@cpu_bound
def heavy_computation(n):
# implementationException handling
Original (sequential) tracebacks are preserved. That should make debugging easier. However, don’t try to catch exceptions. You better want to exit and see them. When you force evaluation potential exceptions will be raised.
Advanced Feature: Force Specific Type of Execution
If you really need more control over the type of execution, use fork.process or fork.thread. They work just like fork.fork but enforce the corresponding type of background execution.
import pkg_resources
for worker_function in pkg_resources.iter_entry_points(group='worker'):
process(worker_function)Advanced Feature: Multiple Execution At Once
You can shorten your programs by using fork.map. It works like fork.fork but submits a function multiple times for each item given by an iterable.
results = fork.map(create_thumbnail, images)fork.map_process and fork.map_thread work accordingly and force a specific type of execution. Use those if really necessary. Otherwise, just use fork.map. fork take care for you in this case again.
In order to wait for the completion of a set of result proxies, use fork.await_all. If you want to unblock by the first unblocking result proxy, call fork.await_any.
There are also blocking variants available: fork.block_map, fork.block_map_process and fork.block_map_thread; in case you need some syntactic sugar:
fork.await_all(fork.map(create_thumbnail, images))
# equals
fork.block_map(create_thumbnail, images)Conclusion
Good
easy to give it a try / easy way from sequential to parallel and back
results evaluate lazily
sequential tracebacks are preserved
it’s thread-safe / cascading forks possible
compatible with Python 2 and 3
Bad
weird calling syntax (no syntax support)
type(result) == ResultProxy
not working with lambdas due to PickleError
needs fix:
not working with coroutines (asyncio) yet (working on it)
cannot fix efficiently:
exception handling (force evaluation when entering and leaving try blocks)
ideas are welcome :-)
1 maintainer
Metadata
1 maintainer
Release files for xfork 0.37
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| xfork-0.37.tar.gz | 6.4 kB | Details |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| xfork-0.37-py3-none-any.whl | Python 3 | none | any | Details |
| xfork-0.37-py2-none-any.whl | Python 2 | none | any | Details |
Total release size: 16.8 kB
Release files / xfork-0.37.tar.gz
| Download URL | xfork-0.37.tar.gz |
|---|---|
| Size | 6.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9ce229ae61ddde063d93907897be8442dd31aaf5f0dcec6269089577af2b307f
|
|
BLAKE2b-256 checksum How to use checksums |
6f2ea91a0d5bd83fd48696b729302f6afe9be5420a34829fb456760ac6ec63e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / xfork-0.37-py3-none-any.whl
| Download URL | xfork-0.37-py3-none-any.whl |
|---|---|
| Size | 5.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bec5f12a11746a2ced266348a072d367a159d03056d0dc8d39fa0211b81f8134
|
|
BLAKE2b-256 checksum How to use checksums |
8e1515aa858489048267da37dfc6c3af67ac75ce32bd63f913b270b15db5bf43
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / xfork-0.37-py2-none-any.whl
| Download URL | xfork-0.37-py2-none-any.whl |
|---|---|
| Size | 5.2 kB |
| Tags | Python 2 |
|
SHA-256 checksum How to use checksums |
1765040264552879c75ea501e3bada2f57627dc1497d4d7579a9ede4c69543a4
|
|
BLAKE2b-256 checksum How to use checksums |
32989c137e49d3fa9fb3cb7d96cc89ac4aa63846d0907309ad9e0843aab692cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
