Replacement robots.txt Parser
Project description
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
- Memoization of fetched robots.txt
- Expiration taken from the Expires header
- Batch queries
- Configurable user agent for fetching robots.txt
- Automatic refetching basing on expiration
Release history Release notifications
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size & hash SHA256 hash help | File type | Python version | Upload date |
|---|---|---|---|
| reppy-0.3.1.tar.gz (7.7 kB) Copy SHA256 hash SHA256 | Source | None |
