Create async web requests in no time
Project description
🔀 Unparallel
With Unparallel you can easily create thousands of web requests in an efficient way leveraging Python's async capabilities.
Unparallel is built on top of HTTPX and aims to support its rich set of features.
Installation
pip install unparallel
Example
A simple example of doing several GET requests to an HTTP web service:
import asyncio
from unparallel import up
async def main():
urls = [f"https://httpbin.org/get?i={i}" for i in range(5)]
results = await up(urls)
print([item["args"] for item in results])
asyncio.run(main())
This prints:
Making async requests: 100%|███████████| 5/5 [00:00<00:00, 9.98it/s]
[{'i': '0'}, {'i': '1'}, {'i': '2'}, {'i': '3'}, {'i': '4'}]
Similarly, we can do a bunch of POST requests. This time we will use a single path but multiple payloads:
import asyncio
from unparallel import up
async def main():
url = "https://httpbin.org/post"
payloads = [{"obj_id": i} for i in range(5)]
results = await up(url, method="post", payloads=payloads)
print([item["data"] for item in results])
asyncio.run(main())
This prints:
Making async requests: 100%|███████████| 5/5 [00:00<00:00, 9.98it/s]
['{"obj_id": 0}', '{"obj_id": 1}', '{"obj_id": 2}', '{"obj_id": 3}', '{"obj_id": 4}']
For more details on the usage and examples, check out the docs.
Why unparallel? Why async?
Async is a really powerful feature - especially when you have to wait for I/O.
Here is an example of making 20 web requests synchronously vs. asynchronously via unparallel
.
As you can see, the async version finishes in less than a second.
Code for sync
import httpx
from tqdm import tqdm
def main():
url = "https://httpbin.org"
paths = [f"/get?i={i}" for i in range(20)]
results = [
httpx.get(f"{url}{path}") for path in tqdm(paths, desc="Making sync requests")
]
assert len(results) == 20
if __name__ == "__main__":
main()
Code for async
import asyncio
from unparallel import up
async def main():
url = "https://httpbin.org"
paths = [f"/get?i={i}" for i in range(20)]
results = await up(paths, base_url=url)
assert len(results) == 20
if __name__ == "__main__":
asyncio.run(main())
Contributing
As this project is still in early development, I'm happy for any feedback and contributions! Please refer to the contributing guidelines for details.
License
This project is licensed under the terms of the MIT
license. See LICENSE for more details.
Credits
This project was heavily inspired by the blog post Making 1 million requests with python-aiohttp by Paweł Miech.
I created this project with python-package-template.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for unparallel-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18ced43c05e7267036670c3b3a00ccd5977f5284dcb3d9e1443950c1a0cdf5b3 |
|
MD5 | 571dfc3c1eac627ec34ab930839b99d0 |
|
BLAKE2b-256 | aa237f93644df891489127e50a0e539f92dbb965001a2d63ec8174b1df40a6da |