Bounded Process&Thread Pool Executor
Project description
Bounded Process&Thread Pool Executor
BoundedSemaphore for ProcessPoolExecutor & ThreadPoolExecutor from concurrent.futures
Installation
pip install bounded-pool-executor
What is the main problem?
If you use the standard module "concurrent.futures" and want to simultaneously process several million data, then a queue of workers will take up all the free memory.
If the script is run on a weak VPS, this will lead to a memory leak.
BoundedProcessPoolExecutor VS ProcessPoolExecutor
BoundedProcessPoolExecutor
BoundedProcessPoolExecutor will put a new worker in queue only when another worker has finished his work.
from bounded_pool_executor import BoundedProcessPoolExecutor
from time import sleep
from random import randint
def do_job(num):
sleep_sec = randint(1, 10)
print('value: %d, sleep: %d sec.' % (num, sleep_sec))
sleep(sleep_sec)
with BoundedProcessPoolExecutor(max_workers=5) as worker:
for num in range(10000):
print('#%d Worker initialization' % num)
worker.submit(do_job, num)
Result:
Classic concurrent.futures.ProcessPoolExecutor
ProcessPoolExecutor inserts all workers into the queue and expects tasks to be performed as the new worker is released, depending on the value of max_workers
.
import concurrent.futures
from time import sleep
from random import randint
def do_job(num):
sleep_sec = randint(1, 3)
print('value: %d, sleep: %d sec.' % (num, sleep_sec))
sleep(sleep_sec)
with concurrent.futures.ProcessPoolExecutor(max_workers=5) as worker:
for num in range(100000):
print('#%d Worker initialization' % num)
worker.submit(do_job, num)
Result:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for bounded_pool_executor-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e092221bc38ade555e1064831f9ed800580fa34a4b6d8e9dd3cd961549627f6e |
|
MD5 | 15bf4b1c6c69a6d06f49d886d972c7bf |
|
BLAKE2b-256 | 23f1e34501c1228415e9fbcac8cb9c81098900e78331b30eeee1816176324bab |
Close
Hashes for bounded_pool_executor-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f164d64919db1e6a5c187cce281f62bc559a5fed4ce064942e650c227aef190 |
|
MD5 | e8646f3884ea0bd0d75411f0775bc39f |
|
BLAKE2b-256 | bc2372ecfe284a1da711257ff310b29c6667d0187a608322d58bf1c7a927c7b2 |