Parallelize the execution of tasks with pytask.
Project description
pytask-parallel
Parallelize the execution of tasks with pytask-parallel which is a plugin for pytask.
Installation
pytask-parallel is available on PyPI and Anaconda.org. Install it with
$ pip install pytask-parallel
# or
$ conda install -c conda-forge pytask-parallel
By default, the plugin uses loky’s robust implementation of the ProcessPoolExecutor.
It is also possible to select the ProcessPoolExecutor or ThreadPoolExecutor from the concurrent.futures module as backends to execute tasks asynchronously.
Usage
To parallelize your tasks across many workers, pass an integer greater than 1 or 'auto' to the command-line interface.
$ pytask -n 2
$ pytask --n-workers 2
# Starts os.cpu_count() - 1 workers.
$ pytask -n auto
Using processes to parallelize the execution of tasks is useful for CPU bound tasks such as numerical computations. (Here is an explanation on what CPU or IO bound means.)
For IO bound tasks, tasks where the limiting factor are network responses, access to files, you can parallelize via threads.
$ pytask --parallel-backend threads
You can also set the options in one of the configuration files (pytask.ini, tox.ini, or setup.cfg).
# This is the default configuration. Note that, parallelization is turned off.
[pytask]
n_workers = 1
parallel_backend = loky # or processes or threads
Warning
It is not possible to combine parallelization with debugging. That is why --pdb or --trace deactivate parallelization.
If you parallelize the execution of your tasks using two or more workers, do not use breakpoint() or import pdb; pdb.set_trace() since both will cause exceptions.
Changes
Consult the release notes to find out about what is new.
Development
pytask-parallel does not call the pytask_execute_task_protocol hook specification/entry-point because pytask_execute_task_setup and pytask_execute_task need to be separated from pytask_execute_task_teardown. Thus, plugins which change this hook specification may not interact well with the parallelization.
There are two PRs for CPython which try to re-enable setting custom reducers which should have been working, but does not. Here are the references.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytask_parallel-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b82ef318a1778e0601b061668c282b58c3450dd090133ebdd701fca080dffa6 |
|
MD5 | 0e376fd8a712f48ee70cce9aacb330e4 |
|
BLAKE2b-256 | ee55eaea3855bf218d22815735bc2c1b9f24af9f297b22e015381086bbcbdbba |