A thread pool that auto-manages the number of worker threads and is compatible with the "futures" module ThreadPoolExecutor interface.
Project description
# CollapsingThreadPoolExecutor
The CollapsingThreadPoolExecutor is inspired by and compatible with the ThreadPoolExecutor from the
"futures" module, it operates differently in that worker threads are handled with a stack which results in the same worker or workers doing all the work (and idle workers being destroyed).
## How to install
$ pip install collapsing-thread-pool-executor
## How to develop
**Prerequisites**
* python3 w/ pip
* python2 w/ pip
* virtualenvwrapper
* entr
**Set up the environments**
$ mkvirtualenv -p `which python2.7` collapsing-thread-pool-executor-py2
$ pip install .
$ pip install -r requirements.txt
# mkvirtualenv -p `which python3` collapsing-thread-pool-executor-py3
$ pip install .
# pip install -r requirements.txt
**Watch the tests**
# watch python2 tests in one window
$ workon collapsing-thread-pool-executor-py2
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor
# watch python3 tests in one window
$ workon collapsing-thread-pool-executor-py3
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor
## Examples
The example below will execute `some_task()` 100 times; as `some_task()` should take a second to execute and as we've allocated 10 workers, the whole thing should take about 10 seconds.
import time
from collapsing_thread_pool_executor import CollapsingThreadPoolExecutor
def some_task():
time.sleep(1)
# all arguments are optional
pool = CollapsingThreadPoolExecutor(
workers=10,
thread_name_prefix='SomePool',
permitted_thread_age_in_seconds=60,
)
for i in range(0, 100):
pool.submit(some_task)
The CollapsingThreadPoolExecutor is inspired by and compatible with the ThreadPoolExecutor from the
"futures" module, it operates differently in that worker threads are handled with a stack which results in the same worker or workers doing all the work (and idle workers being destroyed).
## How to install
$ pip install collapsing-thread-pool-executor
## How to develop
**Prerequisites**
* python3 w/ pip
* python2 w/ pip
* virtualenvwrapper
* entr
**Set up the environments**
$ mkvirtualenv -p `which python2.7` collapsing-thread-pool-executor-py2
$ pip install .
$ pip install -r requirements.txt
# mkvirtualenv -p `which python3` collapsing-thread-pool-executor-py3
$ pip install .
# pip install -r requirements.txt
**Watch the tests**
# watch python2 tests in one window
$ workon collapsing-thread-pool-executor-py2
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor
# watch python3 tests in one window
$ workon collapsing-thread-pool-executor-py3
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor
## Examples
The example below will execute `some_task()` 100 times; as `some_task()` should take a second to execute and as we've allocated 10 workers, the whole thing should take about 10 seconds.
import time
from collapsing_thread_pool_executor import CollapsingThreadPoolExecutor
def some_task():
time.sleep(1)
# all arguments are optional
pool = CollapsingThreadPoolExecutor(
workers=10,
thread_name_prefix='SomePool',
permitted_thread_age_in_seconds=60,
)
for i in range(0, 100):
pool.submit(some_task)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for collapsing-thread-pool-executor-2018.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a266659e273b734b2f2c12f36effe6942841f1b70b25996d9e83c9e752a230ad |
|
MD5 | e44de4534a4fbcc78ba32a9be52083f2 |
|
BLAKE2b-256 | 4e45eb0b0a9c3db7c23e4945fd4eedc0aa45d9df7c23cbd05d8b240786e84de2 |
Close
Hashes for collapsing_thread_pool_executor-2018.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8edd559c5f3c433f593b5f179fce1a39da25273fbd04f1100a43ce6afe168381 |
|
MD5 | 8a665d3bb283e7bc164ff3898864d899 |
|
BLAKE2b-256 | f2e2886fe27ad887b1105e5d7024bf56528447a2a43efe42c0ad0994e2c92e04 |