parallel-execute
Lightweight Python wrappers for easy multiprocessing and multithreading.
Run multiple functions in parallel using a simple API built on top of threading or multiprocessing. Supports error handling, result tracking, timeouts, and controlled concurrency.
Installation
Install the package using pip:
pip install parallel-execute
Usage Example
1. Create a Loom
A Loom takes multiple tasks (functions) and executes them in parallel using either threads or processes.
Using Threads:
from pexecute.thread import ThreadLoom
loom = ThreadLoom(max_runner_cap=10)
Using Processes:
from pexecute.process import ProcessLoom
loom = ProcessLoom(max_runner_cap=10)
max_runner_cap: The maximum number of threads/processes to run in parallel. You can queue as many functions as needed; only max_runner_cap will run at the same time.
2. Add Tasks to the Loom
Add a Single Task:
Use add_function to add individual functions:
loom.add_function(f1, args1, kw1)
loom.add_function(f2, args2, kw2)
loom.add_function(f3, args3, kw3)
Add Multiple Tasks at Once:
Use add_work to add a batch of functions:
work = [(f1, args1, kwargs1), (f2, args2, kwargs2), (f3, args3, kwargs3)]
loom.add_work(work)
3. Execute Tasks
Once all tasks are added, call execute() to run them. It returns a dictionary mapping each task to its result.
output = loom.execute()
By default, the keys are integers in the order the functions were added. Each value is a dictionary containing the result and execution metadata.
Example:
def fun1():
return "Hello World"
def fun2(a):
return a
def fun3(a, b=0):
return a + b
loom.add_function(fun1, [], {})
loom.add_function(fun2, [1], {})
loom.add_function(fun3, [1], {'b': 3})
output = loom.execute()
>>> output
{
0: {
'output': 'Hello World',
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
'execution_time': 0.001498,
},
1: {
'output': 1,
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
'execution_time': 0.001061
},
2: {
'output': 4,
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
'execution_time': 0.001426
}
}
4. Using Custom Keys
You can assign a custom key to each task. This allows you to identify results more explicitly in the output dictionary.
loom.add_function(fun1, [], {}, 'key1')
loom.add_function(fun2, [1], {}, 'fun2')
loom.add_function(fun3, [1], {'b': 3}, 'xyz')
output = loom.execute()
>>> output
{
'key1': {
'output': 'Hello World',
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
'execution_time': 0.001498,
},
'fun2': {
'output': 1,
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
'execution_time': 0.001061
},
'xyz': {
'output': 4,
'got_error': False,
'error': None,
'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
'execution_time': 0.001426
}
}
Migration Notice
parallel-execute is now powered by a newer, more powerful backend called concurra.
New users are encouraged to switch to the new interface:
from concurra import TaskRunner
runner = TaskRunner()
runner.add_task(my_func, *args, **kwargs)
results = runner.run()
Backward compatibility with ThreadLoom and ProcessLoom is currently maintained, but may be deprecated in future versions.
Metadata
Release files for parallel-execute 2.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| parallel_execute-2.0.3.tar.gz | 5.2 kB | Without provenance | Details for parallel_execute-2.0.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| parallel_execute-2.0.3-py3-none-any.whl | Python 3 | none | any | Details for parallel_execute-2.0.3-py3-none-any.whl |
Total release size: 11.5 kB
Release files / parallel_execute-2.0.3.tar.gz
| Download URL | parallel_execute-2.0.3.tar.gz |
|---|---|
| Size | 5.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a8deae6233b657733036ea8632a7a7525a24169d90a94da533d64f7bdf157276
|
|
BLAKE2b-256 checksum How to use checksums |
31c2c5643d1ed5ce6f867566db55ca9a4a925692e3b31b4310f4f9bcea9185fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.5
|
Release files / parallel_execute-2.0.3-py3-none-any.whl
| Download URL | parallel_execute-2.0.3-py3-none-any.whl |
|---|---|
| Size | 6.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
57e574fe5f1d7c7879375712087beb4ce3de6282ed84044fb15eeaa523854d2c
|
|
BLAKE2b-256 checksum How to use checksums |
00f2d4ca09aa00eb778c75cb4e90cd42e56f5af98cf22221e19aaea7f8c801e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.5
|
Security for parallel-execute 2.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.