robots-checker 1.2.3

A tool to filter out data from robots.txt restricted URL domains.

robots-checker

This is a package for convenient robots.txt based compliant filtering. By compliance filtering, we evaluates robots.txt rules specifically for AI training user agents, as shown in the list below.

"AI2Bot",                       # AI2  
"Applebot-Extended",            # Apple  
"Bytespider",                   # Bytedance  
"CCBot",                        # Common Crawl  
"CCBot/2.0",                    # Common Crawl  
"CCBot/1.0",                    # Common Crawl  
"ClaudeBot",                    # Anthropic  
"cohere-training-data-crawler", # Cohere  
"Diffbot",                      # Diffbot  
"Meta-ExternalAgent",           # Meta  
"Google-Extended",              # Google  
"GPTBot",                       # OpenAI  
"PanguBot",                     # Huawei  
"*"

Installation

Currently, only robots.txt checking as of January 2025 is supported.

Install the package

pip install robots-checker==1.2.2

Usage

import url_checker
checker = url_checker.RobotsTxtComplianceChecker() # "Jan-2025"
status = checker.is_compliant("https://blog.example.com/some-page")
print(status)   # ➜  "Compliant"  or  "NonCompliant"

More?

For more information, please check our 🕸️ website