Pure-Python robots.txt parser with support for modern conventions

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for anubhavp from gravatar.com anubhavp Avatar for Gallaecio from gravatar.com Gallaecio Avatar for scrapy from gravatar.com scrapy Avatar for whalebot_helmsman from gravatar.com whalebot_helmsman

Unverified details

These details have not been verified by PyPI
Meta
Classifiers
Report project as malware

Project description

Supported Python Versions CI

Protego is a pure-Python robots.txt parser with support for modern conventions.

Install

To install Protego, simply use pip:

pip install protego

Usage

>>> from protego import Protego
>>> robotstxt = """
... User-agent: *
... Disallow: /
... Allow: /about
... Allow: /account
... Disallow: /account/contact$
... Disallow: /account/*/profile
... Crawl-delay: 4
... Request-rate: 10/1m                 # 10 requests every 1 minute
...
... Sitemap: http://example.com/sitemap-index.xml
... Host: http://example.co.in
... """
>>> rp = Protego.parse(robotstxt)
>>> rp.can_fetch("http://example.com/profiles", "mybot")
False
>>> rp.can_fetch("http://example.com/about", "mybot")
True
>>> rp.can_fetch("http://example.com/account", "mybot")
True
>>> rp.can_fetch("http://example.com/account/myuser/profile", "mybot")
False
>>> rp.can_fetch("http://example.com/account/contact", "mybot")
False
>>> rp.crawl_delay("mybot")
4.0
>>> rp.request_rate("mybot")
RequestRate(requests=10, seconds=60, start_time=None, end_time=None)
>>> list(rp.sitemaps)
['http://example.com/sitemap-index.xml']
>>> rp.preferred_host
'http://example.co.in'

Using Protego with Requests:

>>> from protego import Protego
>>> import requests
>>> r = requests.get("https://google.com/robots.txt")
>>> rp = Protego.parse(r.text)
>>> rp.can_fetch("https://google.com/search", "mybot")
False
>>> rp.can_fetch("https://google.com/search/about", "mybot")
True
>>> list(rp.sitemaps)
['https://www.google.com/sitemap.xml']

Comparison

The following table compares Protego to the most popular robots.txt parsers implemented in Python or featuring Python bindings:

Protego

RobotFileParser

Reppy

Robotexclusionrulesparser

Implementation language

Python

Python

C++

Python

Reference specification

Google

Martijn Koster’s 1996 draft

Wildcard support

✓

✓

✓

Length-based precedence

✓

✓

Performance

+40%

+1300%

-25%

API Reference

Class protego.Protego:

Properties

  • sitemaps {list_iterator} A list of sitemaps specified in robots.txt.

  • preferred_host {string} Preferred host specified in robots.txt.

Methods

  • parse(robotstxt_body) Parse robots.txt and return a new instance of protego.Protego.

  • can_fetch(url, user_agent) Return True if the user agent can fetch the URL, otherwise return False.

  • crawl_delay(user_agent) Return the crawl delay specified for the user agent as a float. If nothing is specified, return None.

  • request_rate(user_agent) Return the request rate specified for the user agent as a named tuple RequestRate(requests, seconds, start_time, end_time). If nothing is specified, return None.

  • visit_time(user_agent) Return the visit time specified for the user agent as a named tuple VisitTime(start_time, end_time). If nothing is specified, return None.

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for anubhavp from gravatar.com anubhavp Avatar for Gallaecio from gravatar.com Gallaecio Avatar for scrapy from gravatar.com scrapy Avatar for whalebot_helmsman from gravatar.com whalebot_helmsman

Unverified details

These details have not been verified by PyPI
Meta
Classifiers

Release history Release notifications | RSS feed

This version

0.6.0

0.5.0

0.4.0

0.3.1

0.3.0

0.2.1

0.2.0

0.1.16

0.1.15

0.1.14

0.1.12

0.1

0.1.dev0 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

protego-0.6.0.tar.gz (3.1 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

protego-0.6.0-py3-none-any.whl (10.3 kB view details)

Uploaded Python 3

File details

Details for the file protego-0.6.0.tar.gz.

File metadata

  • Download URL: protego-0.6.0.tar.gz
  • Upload date:
  • Size: 3.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for protego-0.6.0.tar.gz
Algorithm Hash digest
SHA256 3466f41438421cf90008e98534d5fde47dc16a17482571d021143ac18b70ace9
MD5 c70f1395f80f39a18f6588f5eae721e4
BLAKE2b-256 07a7955c422611d00a6e4a06d30b367ea9bb4fb09d48552e92aef1ba312493c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for protego-0.6.0.tar.gz:

Publisher: publish.yml on scrapy/protego

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file protego-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: protego-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 10.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for protego-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7210e6e06a8db839502baf1bfbcb810689a58e394d31408ef1ef9e4e3d79fc44
MD5 448f37e3f58c512ffba26d861014dbce
BLAKE2b-256 d88cf4dd590f48addf31398f78a78962eaa99eb4c87ac09c1927497032644731

See more details on using hashes here.

Provenance

The following attestation bundles were made for protego-0.6.0-py3-none-any.whl:

Publisher: publish.yml on scrapy/protego

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page