An asynchronous HTTP and RESTful API requests framework for asyncio and Python
Project description
aiorequestful
An asynchronous HTTP and RESTful API requests framework for asyncio and Python
- Full implementation of authorisation handling for authorising with any HTTP service, including OAuth2 flows
- Automatic response payload caching and cache retrieval on a per-endpoint basis to allow fine control over how and when response data is cached
- Automatic payload response handling to transform responses before returning and caching
- Automatic handling of common HTTP error status codes to ensure guaranteed successful requests
- Formulaic approach to retries and backoff handling to ensure smooth requests on sensitive services to handle 'Too Many Requests' style errors
Contents
- Getting Started
- Currently Supported
- Motivation and Aims
- Release History
- Contributing and Reporting Issues
NOTE:
This readme provides a brief overview of the program. Read the docs for full reference documentation.
Installation
Install through pip using one of the following commands:
pip install aiorequestful
python -m pip install aiorequestful
There are optional dependencies that you may install for optional functionality. For the current list of optional dependency groups, read the docs
Getting Started
These quick guides will help you get set up and going with aiorequestful in just a few minutes. For more detailed guides, check out the documentation.
Ultimately, the core part of this whole package is the RequestHandler
.
This object will handle, amongst other things, these core processes:
- creating sessions
- sending requests
- processing responses as configured
- handling error responses including backoff/retry/wait time
- authorising if configured
- caching responses if configured
Each part listed above can be configured as required. Before we get to that though, let's start with a simple example.
Sending simple requests
import asyncio
from typing import Any
from yarl import URL
from aiorequestful.request.handler import RequestHandler
async def send_get_request(handler: RequestHandler, url: str | URL) -> Any:
"""Sends a simple GET request using the given ``handler`` for the given ``url``."""
async with handler:
payload = await handler.get(url)
return payload
request_handler: RequestHandler = RequestHandler.create()
api_url = "https://official-joke-api.appspot.com/jokes/programming/random"
task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)
print(result)
print(type(result).__name__)
And to send many requests, we simply do the following.
async def send_get_requests(handler: RequestHandler, url: str | URL, count: int = 20) -> tuple[Any]:
async with handler:
payloads = await asyncio.gather(*[handler.get(url) for _ in range(count)])
return payloads
results = asyncio.run(send_get_requests(request_handler, url=api_url, count=20))
for result in results:
print(result)
Here, we request some data from an open API that requires no authentication to access. Notice how the data type of the object we retrieve is a string, but we can see from the print that this is meant to be JSON data.
Handling the response payload
When we know the data type we want to retrieve, we can assign a PayloadHandler
to the RequestHandler
to retrieve the data type we require.
from aiorequestful.response.payload import JSONPayloadHandler
payload_handler = JSONPayloadHandler()
request_handler.payload_handler = payload_handler
task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)
print(result)
print(type(result).__name__)
By doing so, we ensure that our RequestHandler
only returns data in a format that we expect.
The JSONPayloadHandler
is set to fail if the data given to it is not valid JSON data.
NOTE: For more info on payload handling, read the docs.
Authorising with the service
Usually, most REST APIs require a user to authenticate and authorise with their services before making any requests.
We can assign an Authoriser
to the RequestHandler
to handle authorising for us.
from aiorequestful.auth.basic import BasicAuthoriser
async def auth_and_send_get_request(handler: RequestHandler, url: str) -> Any:
"""Authorise the ``handler`` with the service before sending a GET request to the given ``url``."""
async with handler:
await handler.authorise()
payload = await handler.get(url)
return payload
authoriser = BasicAuthoriser(login="username", password="password")
request_handler.authoriser = authoriser
task = auth_and_send_get_request(request_handler, url=api_url)
result = asyncio.run(task)
print(result)
NOTE: For more info on authorising including other types of supported authorisation flows, read the docs.
Caching responses
When requesting a large amount of requests from a REST API, you will often find it is comparatively slow for it to respond.
You may add a ResponseCache
to the RequestHandler
to cache the initial responses from
these requests.
This will help speed up future requests by hitting the cache for requests first and returning any matching response
from the cache first before making a HTTP request to get the data.
from aiorequestful.cache.backend import SQLiteCache
cache = SQLiteCache.connect_with_in_memory_db()
request_handler = RequestHandler.create(cache=cache)
task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)
print(result)
However, this example will not cache anything as we have not set up repositories for the endpoints we require. Check out the documentation on caching for more info on setting up cache repositories.
NOTE: We cannot dynamically assign a cache to a instance of
RequestHandler
. Hence, we always need to supply theResponseCache
when instantiating theRequestHandler
.
NOTE: For more info on setting a successful cache and other supported cache backends, read the docs.
Handling error responses
Often, we will receive error responses that we will need to handle.
We can have the RequestHandler
handle these responses by assigning StatusHandler
objects.
from aiorequestful.response.status import ClientErrorStatusHandler, UnauthorisedStatusHandler, RateLimitStatusHandler
response_handlers = [
UnauthorisedStatusHandler(), RateLimitStatusHandler(), ClientErrorStatusHandler()
]
request_handler.response_handlers = response_handlers
task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)
print(result)
print(type(result).__name__)
NOTE: For more info on
StatusHandler
and how they handle each response type, read the docs.
Managing retries and backoff time
Another way we can ensure a successful response is to include a retry and backoff time management strategy.
The RequestHandler
provides two key mechanisms for these operations:
- The
wait_timer
manages the time to wait after every request whether successful or not. This is object-bound i.e. any increase to this timer affects future requests. - The
retry_timer
manages the time to wait after each unsuccessful and unhandled request. This is request-bound i.e. any increase to this timer only affects the current request and not future requests.
Retries and unsuccessful backoff time
As an example, if we want to simply retry the same request 3 times without any backoff time in-between each request, we can set the following.
from aiorequestful.timer import StepCountTimer
request_handler.retry_timer = StepCountTimer(initial=0, count=3, step=0)
We set the count
value to 3
for 3 retries and all other values to 0
to ensure there is no wait time between
these retries.
Should we wish to add some time between each retry, we can do the following.
request_handler.retry_timer = StepCountTimer(initial=0, count=3, step=0.2)
This will now add 0.2 seconds between each unsuccessful request, waiting 0.6 seconds before the final retry for example.
This timer is generated as new for each new request so any increase in time does not carry through to future requests.
Wait backoff time
We may also wish to handle wait time after all requests. This can be useful for sensitive services that often return 'Too Many Requests' errors when making a large volume of requests at once.
from aiorequestful.timer import StepCeilingTimer
request_handler.wait_timer = StepCeilingTimer(initial=0, final=1, step=0.1)
This timer will increase by 0.1 seconds each time it is increased up to a maximum of 1 second.
WARNING: The
RequestHandler
is not responsible for handling when this timer is increased. AStatusHandler
should be used to increase this timer such as theRateLimitStatusHandler
which will increase this timer every time a 'Too Many Requests' error is returned.
This timer is the same for each new request so any increase in time does carry through to future requests.
NOTE: For more info on the available
Timer
objects, read the docs.
Currently Supported
- Cache Backends:
SQLiteCache
- Basic Authorisation:
BasicAuthoriser
- OAuth2 Flows:
AuthorisationCodeFlow
AuthorisationCodePKCEFlow
ClientCredentialsFlow
Motivation and Aims
The key aim of this package is to provide a performant and extensible framework for interacting with REST API services and other HTTP frameworks.
As a new developer, I found it incredibly confusing understanding the myriad ways one can authenticate with a REST API, which to select for my use case, how to implement it in code and so on. I then found it a great challenge learning how to get the maximum performance from my applications for HTTP requests while balancing this against issues when accessing sensitive services which often return 'Too Many Requests' type errors as I improved the performance of my applications. As such, I separated out all the code relating to HTTP requests into this package so that other developers can use what I have learned in their applications too.
This package should implement the following:
- all possible authorisation flows for these types of services
- intelligent caching per endpoint for these responses to many common and appropriate cache backends to allow for:
- storing of responses in a
- reduction in request-response times by retrieving responses from the cache instead of HTTP requests
- reducing load on sensitive HTTP-based services by hitting the cache instead, thereby reducing 'Too Many Requests' type errors
- automatic handling of common HTTP error status codes to ensure guaranteed successful requests
- other quality of life additions to ensure a large volume of responses are returned in the fastest possible time e.g. backoff/retry/wait timers
In so doing, I hope to make the access of data from these services as seamless as possible and provide the foundation of this part of the process in future applications and use cases.
Release History
For change and release history, check out the documentation.
Contributing and Reporting Issues
If you have any suggestions, wish to contribute, or have any issues to report, please do let me know via the issues tab or make a new pull request with your new feature for review.
For more info on how to contribute to aiorequestful, check out the documentation.
I hope you enjoy using aiorequestful!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aiorequestful-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b404749283c370012956b8e8f06cce12385fd5dd96a7fec6e3fe6999adfb495e |
|
MD5 | f4057f84b8a8734ddbd057421b0b0c0c |
|
BLAKE2b-256 | 33a945f306f3eaa91f7c885933e00eb097c3d58dabd9e9f578b1e3b3bf896a12 |