Bounded Process&Thread Pool Executor
Project description
Bounded Process&Thread Pool Executor
BoundedSemaphore for ProcessPoolExecutor & ThreadPoolExecutor from concurrent.futures
What is the main problem?
If you use the standard module “concurrent.futures” and want to simultaneously process several million data, then a queue of workers will take up all the free memory.
If the script is run on a weak VPS, this will lead to a memory leak.
BoundedProcessPoolExecutor VS ProcessPoolExecutor
BoundedProcessPoolExecutor
BoundedProcessPoolExecutor will put a new worker in queue only when another worker has finished his work.
from bounded_pool_executor import BoundedProcessPoolExecutor
from time import sleep
from random import randint
def do_job(num):
sleep_sec = randint(1, 10)
print('value: %d, sleep: %d sec.' % (num, sleep_sec))
sleep(sleep_sec)
with BoundedProcessPoolExecutor(max_workers=5) as worker:
for num in range(10000):
print('#%d Worker initialization' % num)
worker.submit(do_job, num)
Result:
Classic concurrent.futures.ProcessPoolExecutor
ProcessPoolExecutor inserts all workers into the queue and expects tasks to be performed as the new worker is released, depending on the value of max_workers
.
import concurrent.futures
from time import sleep
from random import randint
def do_job(num):
sleep_sec = randint(1, 3)
print('value: %d, sleep: %d sec.' % (num, sleep_sec))
sleep(sleep_sec)
with concurrent.futures.ProcessPoolExecutor(max_workers=5) as worker:
for num in range(100000):
print('#%d Worker initialization' % num)
worker.submit(do_job, num)
Result:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for bounded_pool_executor-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19807a6ea4e2cac322bb69bfcf19949c9e33ee3b43e06056e6b1a3e488e7df89 |
|
MD5 | 9363c2a4cd8d5f8f2d476a6de1a29d33 |
|
BLAKE2b-256 | d8d62ca7a52fd5e741a6127118d3f997d7685f29f728f7db711a3aca65278640 |