Skip to main content

CoroutinePoolExecutor compatible to ThreadPoolExecutor and ProcessPoolExecutor

Project description

Though CoExecutor uses 1 CPU core at any time,

limitations such as ‘max_workers’ are needed for limiting concurrent asyncio connections.

This is useful when you are building a asyncio-based web crawler.

It is designed to have exactly same result with respect to python’s ThreadPoolExecutor and ProcessPoolExecutor.

Also, CoExecutor supports additional features like ‘limit’ or ‘out_of_order’.

coexecutor.map(…, limit=X) limits the number of pending futures (done, but not collected yet).

coexecutor.map(…, out_of_order=True) ignores the input iterator’s ordering.

The results are collected by the termination order of the jobs.

CoExecutor constructor requires a loop to rely on.

With debug=True, it will print the number of active coroutines per second.

Works with Python3.5+.

Project details


Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page