Skip to main content

Extending Python's process pool to support async functions.

Project description

aio-pool

Extending Python's multiporcessing.Pool to support coroutine functions.
Can be useful for when using a server with very high bandwidth or doing both very large IO and CPU tasks at the same time.

All methods of multiprocessing.Pool are supported.
All paramters for multiprocessing.Pool are supported.

examples:

Setting concurrency limit. This means each process can run with up to 8 concurrent tasks at a time.

import asyncio
from aio_pool import AioPool


async def powlong(a):
  await asyncio.sleep(1)
  return a**2

if __name__ == '__main__':
  with AioPool(processes=2, concurrency_limit=8) as pool:
    results = pool.map(powlong, [i for i in range(16)])  # Should take 2 seconds (2*8).
    print(results) 

Async initliazers are also suppported.

import asyncio
from aio_pool import AioPool

async def start(message):
  await asyncio.sleep(1)
  print(message)

async def powlong(a):
  await asyncio.sleep(1)
  return a**2

if __name__ == '__main__':
  with AioPool(processes=2, 
               concurrency_limit=8, 
               initializer=start,
               init_args=("Started with AioPool", )) as pool:
    results = pool.map(powlong, [i for i in range(16)])  # Should take 2 seconds (2*8).
    print(results) 
    

By default, AioPool also set up a default executor for any non-async tasks.
The size can be determined by threadpool_size arguemnt, which defaults to 1.
None default event loops(uvloop for example) are supported as well, using the loop_initializer argument.
Also, non-async functions are supported by default, as the AioPool worker identify if the function is async or not.
If the function is not async, it runs inside the threadpool, to allow the requested concurrency.
This means that order of execution is not guaranteed, even if the function is not async.
However, the order of results is guaranteed through the pool API (map, starmap, apply, etc...).

from aio_pool import AioPool
import uvloop

with AioPool(loop_initializer=uvloop.new_event_loop, threadpool_size=4) pool:
  pool.map(print, [i for i in range(8)])

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aio-pool-0.2.0.tar.gz (5.7 kB view details)

Uploaded Source

Built Distribution

aio_pool-0.2.0-py3-none-any.whl (5.9 kB view details)

Uploaded Python 3

File details

Details for the file aio-pool-0.2.0.tar.gz.

File metadata

  • Download URL: aio-pool-0.2.0.tar.gz
  • Upload date:
  • Size: 5.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.7 CPython/3.9.6 Linux/5.8.0-1036-azure

File hashes

Hashes for aio-pool-0.2.0.tar.gz
Algorithm Hash digest
SHA256 37c4155e5c6e03c8a7c45c84a34cca47f9214c48f1637ecbc7b39a33716ec463
MD5 357b354339f61c88fe7ced7cb44db6e6
BLAKE2b-256 95c324b23a97d666f44bf53b224b19e798673a2c048265cfe30fc5219fbd7ac2

See more details on using hashes here.

Provenance

File details

Details for the file aio_pool-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: aio_pool-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 5.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.7 CPython/3.9.6 Linux/5.8.0-1036-azure

File hashes

Hashes for aio_pool-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e0b609cd83da06312fe50f5e2f75803029be494c9b237e68fe59d585c704e839
MD5 7ff8309f3e272b62dac9fc72af0fa19d
BLAKE2b-256 77108b3e8c8a7c7d3aff1145ebf4c441e3ed29d49d8d4dddb106c0dcf807af64

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page