A Python package to check URL paths against robots directives a robots.txt file.
Project description
Robots Text Processor
This is a Python package to check URL paths against robots directives a robots.txt file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
robotstxt-1.0.3.tar.gz
(6.4 kB
view hashes)