concurrent execution in various flavors
Project description
godale
Debauchery tools for concurrent task execution.
Here you find wrappers around some concurrent.futures
, multiprocessing
and billiard
functions
to facilitate switching between parallelization implementations:
concurrent.futures
multithreading: Python 3 standard tool for multithreadingconcurrent.futures
multiprocessing: Python 3 standard tool for multiprocessingmultiprocessing
multiprocessing: Python 3 standard tool for multiprocessingbilliard
multiprocessing: Multiprocessing forcelery
workers.
All four currently implemented wrappers handle parallel execution of the tasks and yield
result (FinishedTask
) objects having result()
and exception()
methods. This enables
catching of exceptions without cancelling remaining tasks.
Usage:
from godale import Executor
# the first function argument will be replaced by items from the "items" argument below
# subsequent arguments and keyword arguments can be passed on using "fargs" and "fkwargs"
def _worker_function(a, b, some_kwarg=None):
return a * b
# multiprocessing the Python 3 way
executor = Executor(executor="concurrent_processes")
for task in executor.as_completed(
func=_worker_function, # function to be executed
iterable=range(100), # items to be parallelized
fargs=(10, ), # other function arguments
fkwargs={"a_kwarg"=True} # function keyword arguments
):
try:
print(task.result())
except ValueError:
print("task failed")
# multithreading the Python 3 way
executor = Executor(executor="concurrent_threads")
for task in executor.as_completed(
func=_worker_function,
iterable=range(100),
fargs=(10, )
):
try:
print(task.result())
except ValueError:
print("task failed")
# multiprocessing within a celery worker
executor = Executor(executor="billiard")
for task in executor.as_completed(
func=_worker_function,
iterable=range(100),
fargs=(10, )
):
try:
print(task.result())
except ValueError:
print("task failed")
# using the multiprocessing standard module
executor = Executor(executor="multiprocessing")
for task in executor.as_completed(
func=_worker_function,
iterable=range(100),
fargs=(10, )
):
try:
print(task.result())
except ValueError:
print("task failed")
# use different start_method than "fork"
# NOTE: with concurrent.futures and Python 3.6 and earlier, "start_method" other than
# "fork" will raise an RuntimeError
for task in executor.as_completed(
func=_worker_function,
iterable=range(100),
fargs=(10, ),
start_method="spawn"
):
try:
print(task.result())
except ValueError:
print("task failed")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
godale-0.2-py3-none-any.whl
(6.3 kB
view hashes)