Skip to main content

An alternative to Python's robotparser module

Project description

Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.

This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.

Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.

Robotexclusionrulesparser is released under a BSD license.

Project details


Release history Release notifications

History Node

1.7.1

History Node

1.7.0

History Node

1.6.2

This version
History Node

1.6.1

History Node

1.3

History Node

1.2.2

History Node

1.2.1

History Node

1.2

History Node

1.1

History Node

1.0

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page