Replacement robots.txt Parser in pure Python
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Batch queries
Configurable user agent for fetching robots.txt
Automatic refetching basing on expiration
This is a patched fork of the last pure Python version that works on Python 2 and 3.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
reppy2-0.3.6.tar.gz
(72.7 kB
view details)
Built Distribution
reppy2-0.3.6-py3-none-any.whl
(12.2 kB
view details)
File details
Details for the file reppy2-0.3.6.tar.gz.
File metadata
- Download URL: reppy2-0.3.6.tar.gz
- Upload date:
- Size: 72.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e1113935d24f01aaca474b513655660bd1d45dda96b0300163ec32d0c98ca5fc |
|
| MD5 | bf1d531ff152f8ad8d9719e2789ca579 |
|
| BLAKE2b-256 | 5df726cea5ae51939790a449fd70ca2a85376eda1ceaa9970bac18dced21b0ef |
File details
Details for the file reppy2-0.3.6-py3-none-any.whl.
File metadata
- Download URL: reppy2-0.3.6-py3-none-any.whl
- Upload date:
- Size: 12.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b05d6a600834c663a9603658ca3b2f440c9b0353372f728e4dd5f81f37eeb61c |
|
| MD5 | fced09a4f38aa974b995802ce1fcfd67 |
|
| BLAKE2b-256 | d981b5d136977d6b297ff7256a96b171fd15165121888f822d9643ff30d5c5b9 |
