A utility to concurrently make HTTP requests, with an optional rate limit. Powered by asyncio and aiohttp
Project description
Eidolon
A utility to concurrently make HTTP requests, with an optional rate limit. Powered by asyncio and aiohttp.
Installation
Install and update via pip
pip install --U eidolon
A Simple Example
import asyncio
import logging
from eidolon import Parallelizer
from eidolon.api_request import APIRequest
"""
the callback should ideally be an async function, since methods
like .json or .text on the response are async
"""
async def callback(request, response):
print(response.status)
print(await response.text())
"""
Create an APIRequest object. Each object corresponds to a single request, and can take in an optional callback
Everything except request_method and request_url is optional
"""
req = APIRequest(
request_method="GET",
request_url="https://www.google.com",
request_headers={"User-Agent": "Mozilla/5.0"},
request_params={"q": "python"},
request_json={"key": "value"},
request_form_data={"key": "value"},
callback=callback,
)
p = Parallelizer(
logging_level=logging.INFO, # default: logging.ERROR
max_requests_per_second=10, # default: 1
max_retry_attempts=1, # default: 3
seconds_to_sleep_after_rate_limit_error=10, # default: 15
)
# make 100 requests
asyncio.run(p.make_requests([req] * 100))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
eidolon-0.1.4.tar.gz
(4.6 kB
view details)
Built Distribution
File details
Details for the file eidolon-0.1.4.tar.gz
.
File metadata
- Download URL: eidolon-0.1.4.tar.gz
- Upload date:
- Size: 4.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.10.6 Linux/5.15.0-73-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | caa6603ecab27af142fa0f441a616dca69d7e128b77c82a8c0d2bd586f0bc844 |
|
MD5 | c6228768e41f47b711fbc0993fd2f850 |
|
BLAKE2b-256 | d84123ccaaac864a9b7d48553eb10fa13a8d2b87bcdd081130c384ef7473caa0 |
File details
Details for the file eidolon-0.1.4-py3-none-any.whl
.
File metadata
- Download URL: eidolon-0.1.4-py3-none-any.whl
- Upload date:
- Size: 5.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.10.6 Linux/5.15.0-73-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f007bec7cb9549a9b43061d2e08596e6ca020e29592cd1ac762e8a5a13e5c1f |
|
MD5 | 09bbf63e411df65e31a1ddb00aa04ffa |
|
BLAKE2b-256 | fd01aa477d0e506eda28e2376fa39ee27ecc4c06750e4d28cfebe556d03a7ca3 |