asynconsumer is a simple library for processing each items within iterable using asyncio.
Project description
asynconsumer
asynconsumer is a simple library for processing each items within iterable using asyncio.
How to install
$ pip3 install asynconsumer
Quick start
- Define a function or a coroutine function which takes one argument.
- Now, call
async_run(iterable, function)
and you can get list which each items was applied with the function.
concurrency
is an optional parameter can changes number of concurrently executed coroutines(default: 1).
Examples
coro
in the following code is a coroutine function that convert a word passed as an argument to an upper case.
So, async_run(['ham', 'egg', 'spam'], coro, concurrency=2)
will return ['HAM', 'EGG', 'SPAM']
.
>>> from asynconsumer.core import async_run
>>> import asyncio
>>> async def coro(target):
... print('start: {}'.format(target))
... await asyncio.sleep(1)
... print('end: {}'.format(target))
... return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], coro, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>
You can also pass the normal functions.
>>> from asynconsumer.core import async_run
>>> import time
>>> def func(target):
... print('start: {}'.format(target))
... time.sleep(1)
... print('end: {}'.format(target))
... return target.upper()
...
>>> results = async_run(['ham', 'egg', 'spam'], func, concurrency=2)
start: ham
start: egg
end: ham
end: egg
start: spam
end: spam
>>> results
['HAM', 'EGG', 'SPAM']
>>>
More samples.
Sample codes are put in asynconsumer/samples.
Concurrent get HTTP resources
$ pip3 install aiohttp
$ python3 -q
>>> from asynconsumer import fetch_http_resources
>>> urls = ['https://avatars3.githubusercontent.com/u/13819005?s=460&v=4']
>>> fetch_http_resources(urls, '.')
['./8ee20d7c992fee4ac009f4d33c13e276']
>>>
>>> fetch_http_resources(urls=urls, directory='.', naming=lambda url: 'image_{}.jpg'.format(urls.index(url)))
['./image_0.jpg']
Concurrent get GCS objects
$ pip3 install google-cloud-storage
$ python3 -q
>>> from asynconsumer import fetch_gcs_objects
>>> uris = ['gs://...']
>>> fetch_gcs_objects(uris, '.')
['./md5string']
>>>
>>> fetch_http_resources(uris, '.', naming=lambda uri: 'image_{}.jpg'.format(uris.index(uri)))
['./image_0.jpg']
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
File details
Details for the file asynconsumer-1.0.5-py3-none-any.whl
.
File metadata
- Download URL: asynconsumer-1.0.5-py3-none-any.whl
- Upload date:
- Size: 10.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd21108c3fc00ed7b1275e372e5b690329ef82763449320728b39c68f1e062f5 |
|
MD5 | e5d1fb9152961bd2a41bee37ef194310 |
|
BLAKE2b-256 | 01133d83935205a69fa31bad13cb69c90ae25213aaaba8224e4699a97dc5b32c |