Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Batch queries
Configurable user agent for fetching robots.txt
Automatic refetching basing on expiration
This is a patched fork of the last pure Python version that works on Python 2 and 3.
2 maintainers
Metadata
2 maintainers
Release files for reppy2 0.3.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| reppy2-0.3.6.tar.gz | 72.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| reppy2-0.3.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 84.9 kB
Release files / reppy2-0.3.6.tar.gz
| Download URL | reppy2-0.3.6.tar.gz |
|---|---|
| Size | 72.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e1113935d24f01aaca474b513655660bd1d45dda96b0300163ec32d0c98ca5fc
|
|
BLAKE2b-256 checksum How to use checksums |
5df726cea5ae51939790a449fd70ca2a85376eda1ceaa9970bac18dced21b0ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.9
|
Release files / reppy2-0.3.6-py3-none-any.whl
| Download URL | reppy2-0.3.6-py3-none-any.whl |
|---|---|
| Size | 12.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b05d6a600834c663a9603658ca3b2f440c9b0353372f728e4dd5f81f37eeb61c
|
|
BLAKE2b-256 checksum How to use checksums |
d981b5d136977d6b297ff7256a96b171fd15165121888f822d9643ff30d5c5b9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.9
|
