more-executors

This library is intended for use with the concurrent.futures module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

Futures with implicit retry
Futures with implicit cancel on executor shutdown
Futures with implicit cancel after timeout
Futures with transformed output values (sync & async)
Futures resolved by a caller-provided polling function
Throttle the number of futures running at once
Synchronous executor
Bridge concurrent.futures with asyncio
Convenience API for creating executors
Instrumented with Prometheus

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Development

virtualenv and pip may be used to locally install this project from source:

virtualenv ~/dev/python
. ~/dev/python/bin/activate

git clone https://github.com/rohanpm/more-executors
cd more-executors

pip install --editable .

Autotests may be run with pytest:

pip install -r test-requirements.txt
py.test

Submit pull requests against https://github.com/rohanpm/more-executors.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Release files for more-executors 2.11.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for more-executors 2.11.4
File	Size	Uploaded
more-executors-2.11.4.tar.gz	52.3 kB	Feb 1, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for more-executors 2.11.4
File	Interpreter	ABI	Platform	Reset
more_executors-2.11.4-py2.py3-none-any.whl 71.5 kB Feb 1, 2023	Python 2, Python 3	none	any	Details

Total release size: 123.8 kB

Release files / more-executors-2.11.4.tar.gz

Download URL	more-executors-2.11.4.tar.gz
Size	52.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`f1b21d72c4c15069e891d9b96bca05f9abde149e3c11ca54630c5a1a5ee8f4b5`
BLAKE2b-256 checksum How to use checksums	`b1aab16e2197fcf2721e0840ebcd88c687fdf0446608260d6a6d1fcb3e175a8d`
Upload date	Feb 1, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.16`

Release files / more_executors-2.11.4-py2.py3-none-any.whl

Download URL	more_executors-2.11.4-py2.py3-none-any.whl
Size	71.5 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`a304139c6bece5be18aed7dcff4c48440412cb7cbe90f64ba4572772fcb0407f`
BLAKE2b-256 checksum How to use checksums	`d4abd24c17cc6c68125d631a129bf7cc809460106bcce1959aa8a37453ca95d9`
Upload date	Feb 1, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.16`