Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.
This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.
Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.
Robotexclusionrulesparser is released under a BSD license.
1 maintainer
Metadata
1 maintainer
Release files for robotexclusionrulesparser 1.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| robotexclusionrulesparser-1.7.1.tar.gz | 31.5 kB | Details |
Release files / robotexclusionrulesparser-1.7.1.tar.gz
| Download URL | robotexclusionrulesparser-1.7.1.tar.gz |
|---|---|
| Size | 31.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d23aa14ae8145c13c95612d696736bad52a4bd0819ce8c9437ee745098fb8388
|
|
BLAKE2b-256 checksum How to use checksums |
399774634de03a0856160a8c2fa92f03cdf1827c3b1d3d42378d4b79119cd9fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
