Python wrappers for easy multiprocessing and threading
Project description
Python wrappers for easy multiprocessing and threading.
Run multiple functions in parallel using parallel-execute
Installation
pip install parallel-execute
Usage Example
1. Create a loom:
This takes a number of tasks and executes them using a pool of threads/process.
To use threading
from pexecute.thread import ThreadLoom
loom = ThreadLoom(max_runner_cap=10)
To use multiprocessing
from pexecute.process import ProcessLoom
loom = ProcessLoom(max_runner_cap=10)
max_runner_cap: is the number of maximum threads/processes to run at a time. You can add as many as functions you want, but only n functions will run at a time in parallel, n is the max_runner_cap
2. Add tasks in loom
Add a function in loom using add_function
loom.add_function(f1, args1, kw1)
loom.add_function(f2, args2, kw2)
loom.add_function(f3, args3, kw3)
Add multiple functions together using add_work method
work = [(f1, args1, kwargs1), (f2, args2, kwargs2), (f3, args3, kwargs3)]
loom.add_work(work)
3. Execute all tasks
After adding tasks, calling execute will return a dictionary of results corresponding to the keys or the order in which the tasks were added.
output = loom.execute()
key is the order in which the function was added and value is the return data of the function.
# Example:
def fun1():
return "Hello World"
def fun2(a):
return a
def fun3(a, b=0):
return a+b
loom.add_function(fun1, [], {})
loom.add_function(fun2, [1], {})
loom.add_function(fun3, [1], {'b': 3})
output = loom.execute()
>>> output
{0: {'started_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 362899), 'finished_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 364479), 'execution_time': 0.00158, 'output': 'Hello World', 'got_error': False, 'error': None}, 1: {'started_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 364487), 'finished_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 365868), 'execution_time': 0.001381, 'output': 1, 'got_error': False, 'error': None}, 2: {'started_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 366653), 'finished_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 368063), 'execution_time': 0.00141, 'output': 'Hello World', 'got_error': False, 'error': None}, 3: {'started_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 369301), 'finished_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 370670), 'execution_time': 0.001369, 'output': 1, 'got_error': False, 'error': None}, 4: {'started_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 372886), 'finished_time': datetime.datetime(2019, 6, 28, 19, 34, 23, 374007), 'execution_time': 0.001121, 'output': 4, 'got_error': False, 'error': None}}
We can also provide a key to store the function return data.
# Example:
loom.add_function(fun1, [], {}, 'key1')
loom.add_function(fun2, [1], {}, 'fun2')
loom.add_function(fun3, [1], {'b': 3}, 'c')
output = loom.execute()
>>> output
{'key1': {'started_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 208442), 'finished_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 209500), 'execution_time': 0.001058, 'output': 'Hello World', 'got_error': False, 'error': None}, 'fun2': {'started_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 210061), 'finished_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 211147), 'execution_time': 0.001086, 'output': 1, 'got_error': False, 'error': None}, 'c': {'started_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 212602), 'finished_time': datetime.datetime(2019, 6, 28, 19, 36, 18, 213896), 'execution_time': 0.001294, 'output': 4, 'got_error': False, 'error': None}}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for parallel_execute-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d9fcf3973c4494348bd1791923bbdd60fa7d7843e946d279e0c8f1630ddbc31 |
|
MD5 | 494f2d3a34994cce40b32e8707459538 |
|
BLAKE2b-256 | 338db797c2668dd100e2c2fc6cb269b08ab36f5af21743f9360e1e6b30a74366 |