a light multiprocessing/multithreading work dispatcher for python.
Project description
wworks
a light multiprocessing/multithreading work dispatcher for python.
wworks (Wrapped Works) is a work manager that uses both ProcessPoolExecutor and ThreadPoolExecutor to dispatch work by workload over processes and/or threads.
Installation
Check your python version (must be >= 3.7)
> python --version
Python 3.8.12
Install wworks package
> pip install wworks
Usage examples
WorkManager.work(work_name, work_to_do, work_data)
Use WorkManager.work to make multithreading for given function.
from wworks.swworks import WorkManager
def multiply(x, y):
return x*y
# Build 10 tuples from (0, 0) to (9, 9)
work_data = [(x, x) for x in range(10)]
results = WorkManager().work("multiply", multiply, work_data)
for (task_name, task_data, task_result) in results:
print(f"{task_name} : multiply{task_data} => {task_result}")
Task #0 : multiply(0, 0) => 0
Task #1 : multiply(1, 1) => 1
Task #2 : multiply(2, 2) => 4
Task #3 : multiply(3, 3) => 9
Task #4 : multiply(4, 4) => 16
Task #5 : multiply(5, 5) => 25
Task #6 : multiply(6, 6) => 36
Task #7 : multiply(7, 7) => 49
Task #8 : multiply(8, 8) => 64
Task #9 : multiply(9, 9) => 81
In this case, WorkManager create 10 tasks (threads) to process.
WorkManager.chunks(lst, n)
Use WorkManager.chunks to yield n-sized chunks from lst.
from wworks.swworks import WorkManager
# Build 10 tuples from (0, 0) to (9, 9)
work_data = [(x, x) for x in range(10)]
results = WorkManager().chunks(work_data, 4)
for i, chunk in enumerate(results):
print(f"Chunk #{i}")
print(chunk)
Chunk #0
[(0, 0), (1, 1), (2, 2), (3, 3)]
Chunk #1
[(4, 4), (5, 5), (6, 6), (7, 7)]
Chunk #2
[(8, 8), (9, 9)]
In this case, WorkManager yield 4-chunks from all provided tuples.
WorkManager.dispatch(work_to_do, work_data, workload=4)
Use WorkManager.dispatch to make chunked-by-process, multithreading for given function.
from wworks.swworks import WorkManager
def multiply(x, y):
return x*y
# Build 10 tuples from (0, 0) to (9, 9)
work_data = [(x, x) for x in range(10)]
results = WorkManager().dispatch(multiply, work_data)
for (worker_name, worker_result) in results:
print(worker_name)
for (task_name, task_data, task_result) in worker_result:
print(f" - {task_name} : multiply{task_data} => {task_result}")
Worker #0
- Task #0 : multiply(0, 0) => 0
- Task #1 : multiply(1, 1) => 1
- Task #2 : multiply(2, 2) => 4
- Task #3 : multiply(3, 3) => 9
Worker #1
- Task #0 : multiply(4, 4) => 16
- Task #1 : multiply(5, 5) => 25
- Task #2 : multiply(6, 6) => 36
- Task #3 : multiply(7, 7) => 49
Worker #2
- Task #0 : multiply(8, 8) => 64
- Task #1 : multiply(9, 9) => 81
In this case, WorkManager create 3 workers (processes) and give each of them chunked work data respectivelly 4, 4 and 2 tasks to process.
Release History
- 0.1.0
- First version of wworks package
Meta
Mehdi LAKBIR
Distributed under the MIT license. See LICENSE for more information.
https://https://github.com/LMKA/wworks
Contributing
- Fork it (https://github.com/LMKA/wworks/fork)
- Create your feature branch (
git checkout -b feature/fooBar
) - Commit your changes (
git commit -am 'Add some fooBar'
) - Push to the branch (
git push origin feature/fooBar
) - Create a new Pull Request
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.