Robots Text Processor

A Python package for processing and validating robots.txt files according to the Robots Exclusion Protocol (REP) RFC.

Features

• Parse and process robots.txt files
• Extract user-agent rules, allow/disallow directives, and sitemaps
• Test URLs against robots.txt rules
• Validate robots.txt files for compliance with REP RFC
• Generate hashes for content tracking
• Comprehensive error and warning reporting

Installation

pip install robotstxt-package

Usage

Basic Usage

from robotstxt import robots_file

# Create a RobotsFile instance
robots = robots_file("""
User-agent: *
Allow: /
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
""")

# Test a URL
result = robots.test_url("https://example.com/private/page", "*")
print(result)  # {'disallowed': True, 'matching_rule': '/private/'}

# Get sitemaps
for sitemap in robots.sitemaps:
    print(sitemap.url)

Validation

The package includes comprehensive validation of robots.txt files:

from robotstxt import robots_file

# Create a RobotsFile instance
robots = robots_file("""
User-agent: *
Allow: /
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
""")

# Check for validation errors
if robots.has_errors():
    for error in robots.get_validation_errors():
        print(f"Error: {error.message} (Line {error.line_number})")

# Check for validation warnings
if robots.has_warnings():
    for warning in robots.get_validation_warnings():
        print(f"Warning: {warning.message} (Line {warning.line_number})")

The validation system checks for:

• File size exceeding 500KB
• UTF-8 Byte Order Mark (BOM) presence
• Invalid characters in lines
• Proper directive formatting (user-agent, allow, disallow, sitemap)
• Rule block structure (user-agent directives before allow/disallow rules)
• Valid sitemap URLs
• Duplicate rules for the same user-agent
• And more...

Content Tracking

The package provides hashing functionality for tracking changes:

from robotstxt import robots_file

robots = robots_file(content)

# Get content hash
print(robots.hash_raw)  # SHA-256 hash of raw content

# Get rules hash
print(robots.hash_material)  # SHA-256 hash of processed rules

# Get sitemaps hash
print(robots.hash_sitemaps)  # SHA-256 hash of sitemap URLs

Validation Rules

The package validates robots.txt files according to the following rules:

1. File Size

   • Warning if file exceeds 500KB
   • Error if file exceeds 512KB

2. Character Encoding

   • Error if file contains UTF-8 BOM
   • Error if file contains invalid characters

3. Directive Format

   • Error for invalid directive format
   • Error for missing user-agent before allow/disallow rules
   • Warning for common typos in directives

4. Rule Structure

   • Error for duplicate rule blocks
   • Warning for conflicting allow/disallow rules
   • Warning for overly broad rules

5. Sitemaps

   • Error for invalid sitemap URLs
   • Warning for multiple sitemaps

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

compare_robots_files(robots1, robots2)

Compares two robots.txt files and generates a structured diff showing:

• Material differences between the files
• Per-token changes showing added and removed rules
• Sitemap changes

The function normalizes the rules before comparison to ensure accurate diffing regardless of:

• Rule ordering within token groups
• Whitespace differences
• Case sensitivity in tokens
• Duplicate rules

Returns a dictionary containing:

• materially_different: Boolean indicating if the files have different rules
• token_diffs: Dictionary of differences per token
• sitemap_changes: Dictionary of sitemap differences

Example usage

robots1 = RobotsFile(old_content) robots2 = RobotsFile(new_content)

Either way works:

diff = compare_robots_files(robots1, robots2)

or

diff = robots1.compare_with(robots2)

Example output structure:

{

"materially_different": True,

"token_diffs": {

"googlebot": {

"added": ["Allow: /new-path/", "Disallow: /private/"],

"removed": ["Disallow: /old-path/"]

},

"bingbot": {

"added": ["Disallow: /api/"],

"removed": []

}

},

"sitemap_changes": {

"added": ["https://example.com/new-sitemap.xml"],

"removed": ["https://example.com/old-sitemap.xml"]

}

}