Bounded Process&Thread Pool Executor
Project description
Bounded Process&Thread Pool Executor
BoundedSemaphore for ProcessPoolExecutor & ThreadPoolExecutor from concurrent.futures
Installation
pip install bounded-pool-executor
What is the main problem?
If you use the standard module "concurrent.futures" and want to simultaneously process several million data, then a queue of workers will take up all the free memory.
If the script is run on a weak VPS, this will lead to a memory leak.
BoundedProcessPoolExecutor VS ProcessPoolExecutor
BoundedProcessPoolExecutor
BoundedProcessPoolExecutor will put a new worker in queue only when another worker has finished his work.
from bounded_pool_executor import BoundedProcessPoolExecutor from time import sleep from random import randint def do_job(num): sleep_sec = randint(1, 10) print('value: %d, sleep: %d sec.' % (num, sleep_sec)) sleep(sleep_sec) with BoundedProcessPoolExecutor(max_workers=5) as worker: for num in range(10000): print('#%d Worker initialization' % num) worker.submit(do_job, num)
Result:
Classic concurrent.futures.ProcessPoolExecutor
ProcessPoolExecutor inserts all workers into the queue and expects tasks to be performed as the new worker is released, depending on the value of max_workers
.
import concurrent.futures from time import sleep from random import randint def do_job(num): sleep_sec = randint(1, 3) print('value: %d, sleep: %d sec.' % (num, sleep_sec)) sleep(sleep_sec) with concurrent.futures.ProcessPoolExecutor(max_workers=5) as worker: for num in range(100000): print('#%d Worker initialization' % num) worker.submit(do_job, num)
Result:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size bounded_pool_executor-0.0.3-py3-none-any.whl (3.4 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
Filename, size bounded_pool_executor-0.0.3.tar.gz (2.2 kB) | File type Source | Python version None | Upload date | Hashes View |
Close
Hashes for bounded_pool_executor-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f164d64919db1e6a5c187cce281f62bc559a5fed4ce064942e650c227aef190 |
|
MD5 | e8646f3884ea0bd0d75411f0775bc39f |
|
BLAKE2-256 | bc2372ecfe284a1da711257ff310b29c6667d0187a608322d58bf1c7a927c7b2 |
Close
Hashes for bounded_pool_executor-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e092221bc38ade555e1064831f9ed800580fa34a4b6d8e9dd3cd961549627f6e |
|
MD5 | 15bf4b1c6c69a6d06f49d886d972c7bf |
|
BLAKE2-256 | 23f1e34501c1228415e9fbcac8cb9c81098900e78331b30eeee1816176324bab |