Scrape Simple

A web scraper that uses Tor for anonymity and supports text and media extraction.

Features

• Tor integration for anonymous web scraping
• Extract text content from web pages
• Extract media files (images, videos) above a specified size
• Optional Russian text simplification using Natasha
• Optional AI-based image description using BLIP

Installation

pip install scrape-simple

Optional Dependencies

For Russian text simplification:

pip install scrape-simple[russian]

For AI image descriptions:

pip install scrape-simple[ai]

For all features:

pip install scrape-simple[russian,ai]

Usage

Command Line

# Basic usage
scrape-simple https://example.com

# Advanced usage
scrape-simple https://example.com --depth 3 --min-media-size 20480 --simplify-ru --ai-describe-media

Python API

from scrape_simple import WebScraper, SiteContent

# Create scraper
scraper = WebScraper(
    root_url="https://example.com",
    max_depth=2,
    use_existing_tor=True,
    min_media_size=10240,  # 10KB minimum for media files
    simplify_ru=False,
    ai_describe_media=False
)

# Start scraping
site_content = scraper.start()

# Access results
for page in site_content.TextPages:
    print(f"Page: {page.url}, Content length: {len(page.content)}")

for media in site_content.MediaContentList:
    print(f"Media: {media.url}, Type: {media.media_type}, Description: {media.description}")

# Create scraper with media extraction disabled
scraper = WebScraper(
    root_url="https://example.com",
    max_depth=2,
    use_existing_tor=True,
    skip_media=True  # Disable media extraction
)

Requirements

• Python 3.6+
• Tor (must be installed separately)

Command Line Arguments

Anti-Bot Protection Handling

Scrape Simple includes features to bypass common anti-bot protections:

• Browser-like request headers
• Random delays between requests
• Tor IP rotation for 403/429 errors
• Configurable retry mechanism