a simple class wrapper to nice URL requests
Project description
niced_url_request
A simple wrapper class to "nice" url requests in python by setting a maximum frequency to url requests, and using a disk cache.
Motivation
I have needed several times lately to get some relatively large dataset from the net as a series of smaller http requests. This class is a convenient small wrapper for doing so without annoying the server.
Functionalities
- limit request rate
- cache the answers on disk
- perform up to 10 re-tries waiting 10 seconds in between if request fails to make querying large quantities of requests robust to minor network / service interruptions
Installation
pip install niced_url_request
Tests and example
To run tests: pytest -v .
from the present folder.
For a detailed example: see examples
folder.
Small start up example:
from niced_url_request import NicedUrlRequest
niced_requester = NicedUrlRequest()
request_result = niced_requester.perform_request("http://httpbin.org/get?bla1=blabla1")
print(request_result.decode("ASCII"))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
niced_url_request-0.1.tar.gz
(5.4 kB
view hashes)
Built Distribution
Close
Hashes for niced_url_request-0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee8c670bbf8b4d25cbf99e3e57f3737b4c0efa799ea159fe9172b4e47da19276 |
|
MD5 | d6c1f02c836b1731623fe407975c2364 |
|
BLAKE2b-256 | 29a2ad957e30744afa6f8b8f4805547b8a15cceff0f19222804e13ac6d5fc155 |