Replacement robots.txt Parser in pure Python
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Batch queries
Configurable user agent for fetching robots.txt
Automatic refetching basing on expiration
This is a patched fork of the last pure Python version that works on Python 2 and 3.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
reppy2-0.3.5.tar.gz
(10.2 kB
view details)
Built Distribution
File details
Details for the file reppy2-0.3.5.tar.gz.
File metadata
- Download URL: reppy2-0.3.5.tar.gz
- Upload date:
- Size: 10.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6f24d760a938927672ee8037121f45114d44ca653dfd3cbda814ac54eee48e19 |
|
| MD5 | cca03ee3eab8955c9f93bb3e6022e861 |
|
| BLAKE2b-256 | a2620cbdca271e05df5bb37b547b6ccff8191045351fe5b6a9ed58479101305d |
File details
Details for the file reppy2-0.3.5-py2.py3-none-any.whl.
File metadata
- Download URL: reppy2-0.3.5-py2.py3-none-any.whl
- Upload date:
- Size: 12.1 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 17920ce851d8ae22db157cfffb800178cd3b953206300a31db49e20987212d4d |
|
| MD5 | f4d2935cea1573e2d6ebfab8a8f2905f |
|
| BLAKE2b-256 | b190131bef9f8f9280271486dfb49cd776d8c02f5a6d1d9771c68633c0a86e13 |
