reppy2 0.3.6

Replacement robots.txt Parser in pure Python

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

• Memoization of fetched robots.txt

• Expiration taken from the Expires header

• Batch queries

• Configurable user agent for fetching robots.txt

• Automatic refetching basing on expiration

This is a patched fork of the last pure Python version that works on Python 2 and 3.