Skip to main content

asynconsumer is a simple library for processing each items within iterable using asyncio.

Project description

asynconsumer

asynconsumer is a simple library for processing each items within iterable using asyncio.

How to install

$ pip3 install asynconsumer

Quick start

  1. Define a function or a coroutine function which takes one argument.
  2. Now, call async_run(iterable, function) and you can get list which each items was applied with the function.
    concurrency is an optional parameter can changes number of concurrently executed coroutines(default: 1).

Examples

coro in the following code is a coroutine function that convert a word passed as an argument to an upper case.
So, async_run(['ham', 'egg', 'spam'], coro, concurrency=2) will return ['HAM', 'EGG', 'SPAM'].

>>> from asynconsumer.core import async_run
>>> import asyncio
>>> async def coro(target):
...   print('start: {}'.format(target))
...   await asyncio.sleep(1)
...   print('end: {}'.format(target))
...   return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], coro, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>

You can also pass the normal functions.

>>> from asynconsumer.core import async_run
>>> import time
>>> def func(target):
...   print('start: {}'.format(target))
...   time.sleep(1)
...   print('end: {}'.format(target))
...   return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], func, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>

More samples.

Sample codes are put in asynconsumer/samples.

Concurrent get HTTP resources

$ pip3 install aiohttp
$ python3 -q
>>> from asynconsumer import fetch_http_resources
>>> urls = ['https://avatars3.githubusercontent.com/u/13819005?s=460&v=4']
>>> fetch_http_resources(urls, '.')
['./8ee20d7c992fee4ac009f4d33c13e276']
>>>
>>> fetch_http_resources(urls=urls, directory='.', naming=lambda url: 'image_{}.jpg'.format(urls.index(url)))
['./image_0.jpg']

Concurrent get GCS objects

$ pip3 install google-cloud-storage
$ python3 -q
>>> from asynconsumer import fetch_gcs_objects
>>> uris = ['gs://...']
>>> fetch_gcs_objects(uris, '.')
['./md5string']
>>>
>>> fetch_http_resources(uris, '.', naming=lambda uri: 'image_{}.jpg'.format(uris.index(uri)))
['./image_0.jpg']

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for asynconsumer, version 1.0.5
Filename, size File type Python version Upload date Hashes
Filename, size asynconsumer-1.0.5-py3-none-any.whl (10.5 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page