Replacement robots.txt Parser
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
- Memoization of fetched robots.txt
- Expiration taken from the
Expiresheader - Batch queries
- Configurable user agent for fetching robots.txt
- Automatic refetching based on expiration
Release history Release notifications
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size & hash SHA256 hash help | File type | Python version | Upload date |
|---|---|---|---|
| reppy-0.4.10-cp37-cp37m-macosx_10_10_x86_64.whl (134.1 kB) Copy SHA256 hash SHA256 | Wheel | cp37 | |
| reppy-0.4.10.tar.gz (89.4 kB) Copy SHA256 hash SHA256 | Source | None |
