Parallelize the execution of pytask.
Project description
pytask-parallel
Parallelize the execution of tasks with pytask-parallel which is a plugin for pytask.
Installation
pytask-parallel is available on PyPI and Anaconda.org. Install it with
$ pip install pytask-parallel
# or
$ conda config --add channels conda-forge --add channels pytask
$ conda install pytask-parallel
By default, the plugin uses a robust implementation of the ProcessPoolExecutor from loky.
It is also possible to select the ProcessPoolExecutor or ThreadPoolExecutor in the concurrent.futures module as backends to execute tasks asynchronously.
Usage
To parallelize your tasks across many workers, pass an integer greater than 1 or 'auto' to the command-line interface.
$ pytask -n 2
$ pytask --n-workers 2
# Starts os.cpu_count() - 1 workers.
$ pytask -n auto
Using processes to parallelize the execution of tasks is useful for CPU bound tasks such as numerical computations. (Here is an explanation on what CPU or IO bound means.)
For IO bound tasks, tasks where the limiting factor are network responses, accesses to files, you can parallelize via threads.
$ pytask --parallel-backend threads
You can also set the options in one of the configuration files (pytask.ini, tox.ini, or setup.cfg).
# This is the default configuration. Note that, parallelization is turned off.
[pytask]
n_workers = 1
parallel_backend = loky # or processes or threads
Changes
Consult the release notes to find out about what is new.
Development
pytask-parallel does not call the pytask_execute_task_protocol hook specification/entry-point because pytask_execute_task_setup and pytask_execute_task need to be separated from pytask_execute_task_teardown. Thus, plugins which change this hook specification may not interact well with the parallelization.
There are two PRs for CPython which try to re-enable setting custom reducers which should have been working, but does not. Here are the references.
If the TopologicalSorter becomes available for all supported Python versions, deprecate the copied module. Meanwhile, keep it in sync.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytask_parallel-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bac75a5b17faf17ff785d3207981466776ad4f2662ea8f6ea24d05b44a04565a |
|
MD5 | 624e2763f520233ed4af82e027148a01 |
|
BLAKE2b-256 | 99770a003133f9968a2d8c55750c6669d70145fcf22ed7fadb644c9dba48b317 |