Skip to main content

asynconsumer is a simple library for processing each items within iterable using asyncio.

Project description

asynconsumer

asynconsumer is a simple library for processing each items within iterable using asyncio.

How to install

$ pip3 install asynconsumer

Quick start

  1. Define a function or a coroutine function which takes one argument.
  2. Now, call async_run(iterable, function) and you can get list which each items was applied with the function.
    concurrency is an optional parameter can changes number of concurrently executed coroutines(default: 1).

Examples

coro in the following code is a coroutine function that convert a word passed as an argument to an upper case.
So, async_run(['ham', 'egg', 'spam'], coro, concurrency=2) will return ['HAM', 'EGG', 'SPAM'].

>>> from asynconsumer.core import async_run
>>> import asyncio
>>> async def coro(target):
...   print('start: {}'.format(target))
...   await asyncio.sleep(1)
...   print('end: {}'.format(target))
...   return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], coro, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>

You can also pass the normal functions.

>>> from asynconsumer.core import async_run
>>> import time
>>> def func(target):
...   print('start: {}'.format(target))
...   time.sleep(1)
...   print('end: {}'.format(target))
...   return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], func, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>

More samples.

Sample codes are put in asynconsumer/samples.

Concurrent get HTTP resources

$ pip3 install aiohttp
$ python3 -q
>>> from asynconsumer import fetch_http_resources
>>> urls = ['https://avatars3.githubusercontent.com/u/13819005?s=460&v=4']
>>> fetch_http_resources(urls, '.')
['./8ee20d7c992fee4ac009f4d33c13e276']
>>>
>>> fetch_http_resources(urls=urls, directory='.', naming=lambda url: 'image_{}.jpg'.format(urls.index(url)))
['./image_0.jpg']

Concurrent get GCS objects

$ pip3 install google-cloud-storage
$ python3 -q
>>> from asynconsumer import fetch_gcs_objects
>>> uris = ['gs://...']
>>> fetch_gcs_objects(uris, '.')
['./md5string']
>>>
>>> fetch_http_resources(uris, '.', naming=lambda uri: 'image_{}.jpg'.format(uris.index(uri)))
['./image_0.jpg']

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

asynconsumer-1.0.5-py3-none-any.whl (10.5 kB view details)

Uploaded Python 3

File details

Details for the file asynconsumer-1.0.5-py3-none-any.whl.

File metadata

  • Download URL: asynconsumer-1.0.5-py3-none-any.whl
  • Upload date:
  • Size: 10.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for asynconsumer-1.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 fd21108c3fc00ed7b1275e372e5b690329ef82763449320728b39c68f1e062f5
MD5 e5d1fb9152961bd2a41bee37ef194310
BLAKE2b-256 01133d83935205a69fa31bad13cb69c90ae25213aaaba8224e4699a97dc5b32c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page