Skip to main content

CoroutinePoolExecutor compatible to ThreadPoolExecutor and ProcessPoolExecutor

Project description

Though CoExecutor uses 1 CPU core at any time,

limitations such as ‘max_workers’ are needed for limiting concurrent asyncio connections.

This is useful when you are building a asyncio-based web crawler.

It is designed to have exactly same result with respect to python’s ThreadPoolExecutor and ProcessPoolExecutor.

Also, CoExecutor supports additional features like ‘limit’ or ‘out_of_order’.…, limit=X) limits the number of pending futures (done, but not collected yet).…, out_of_order=True) ignores the input iterator’s ordering.

The results are collected by the termination order of the jobs.

CoExecutor constructor requires a loop to rely on.

With debug=True, it will print the number of active coroutines per second.

Works with Python3.5+.

Project details

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page