Replacement robots.txt Parser
Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:
Memoization of fetched robots.txt
Expiration taken from the Expires header
Configurable user agent for fetching robots.txt
Automatic refetching based on expiration
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.