collapsing-thread-pool-executor 2019.11

A thread pool that auto-manages the number of worker threads and is compatible with the "futures" module ThreadPoolExecutor interface.

CollapsingThreadPoolExecutor

The CollapsingThreadPoolExecutor is inspired by and compatible with the ThreadPoolExecutor from the "futures" module, it operates differently in that worker threads are handled with a stack which results in the same worker or workers doing all the work (and idle workers being destroyed).

How to install

$ pip install collapsing-thread-pool-executor

How to develop

Prerequisites

• python3 w/ pip
• python2 w/ pip
• virtualenvwrapper
• entr

Set up the environments

$ mkvirtualenv -p `which python2.7` collapsing-thread-pool-executor-py2
$ pip install .
$ pip install -r requirements.txt

$ mkvirtualenv -p `which python3` collapsing-thread-pool-executor-py3
$ pip install .
$ pip install -r requirements.txt

Watch the tests

# watch python2 tests in one window
$ workon collapsing-thread-pool-executor-py2
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor

# watch python3 tests in one window
$ workon collapsing-thread-pool-executor-py3
$ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor

Examples

The example below will execute some_task() 100 times; as some_task() should take a second to execute and as we've allocated 10 workers, the whole thing should take about 10 seconds.

import time

from collapsing_thread_pool_executor import CollapsingThreadPoolExecutor

def some_task():
    time.sleep(1)

# all arguments are optional
pool = CollapsingThreadPoolExecutor(
    workers=10,
    thread_name_prefix='SomePool',
    permitted_thread_age_in_seconds=60,
)

for i in range(0, 100):
    pool.submit(some_task)