crawlit 0.2.0

Modular, Ethical Python Web Crawler

🕷️ crawlit - Modular, Ethical Python Web Crawler

A powerful, modular, and ethical web crawler built in Python. Designed for security testing, link extraction, and website structure mapping with a focus on clean architecture and extensibility.

🚀 Features

Core Crawling

• Modular Architecture: Easily extend with custom modules and parsers
• Synchronous & Asynchronous: Both sync (Crawler) and async (AsyncCrawler) implementations
• Multi-threaded Support: Thread pool support for concurrent requests in sync crawler
• Depth Control: Set maximum crawl depth to prevent excessive resource usage
• Domain Filtering: Restrict crawling to specific domains or subdomains
• BFS Strategy: Breadth-first search crawling algorithm
• Queue Management: Pause/resume, save/load state, and queue size limits

Ethical Crawling

• Robots.txt Compliance: Configurable robots.txt respect with automatic crawl-delay extraction
• Rate Limiting: Global and per-domain rate limiting with automatic crawl-delay support
• Session Management: Cookie persistence, SSL verification, and custom headers
• Authentication: Support for Basic/Digest Auth, OAuth tokens, API keys, and custom headers
• Retry Logic: Configurable retry attempts with exponential backoff

Content Extraction

• Advanced Table Extraction: Extract tables with support for complex structures and cell spanning
• Image Extraction: Extract and analyze images including alt text and accessibility information
• Keyword Extraction: Identify key terms and phrases from webpage content
• Content Extraction: Optional metadata extraction (title, description, headings, canonical URLs)
• Content Deduplication: Skip pages with duplicate content using SHA-256 hashing

Data Management

• Page Caching: Memory and disk-based caching with TTL support
• Storage Management: Optional HTML content storage (memory or disk-based)
• Crawl Resume: Save and resume interrupted crawls
• Progress Tracking: Real-time progress monitoring with callbacks
• Multiple Output Formats: Export results as JSON, CSV, TXT, or HTML

Advanced Features

• Sitemap Support: Automatic sitemap discovery and parsing from robots.txt
• URL Filtering: Advanced URL filtering (patterns, extensions, query parameters)
• Enhanced Logging: Structured JSON logging, log rotation, configurable levels
• Error Handling: Comprehensive error handling with custom exceptions
• Command Line Interface: Simple, powerful CLI for easy usage
• Programmatic API: Use as a library in your own Python code

📋 Requirements

• Python 3.8+
• Dependencies (will be listed in requirements.txt)

🛠️ Installation

From PyPI (recommended)

# Install the core library
pip install crawlit

# Install with CLI tool support
pip install crawlit[cli]

From Source

# Clone the repository
git clone https://github.com/SwayamDani/crawlit.git
cd crawlit

# Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .

📘 Usage

API Documentation

Full API documentation is available in the docs directory, including documentation for:

• Core crawler modules
• Extraction modules (tables, images, and keywords)
• Output formatters
• Command-line interface

To build and view the documentation:

# Install Sphinx and required packages
pip install sphinx sphinx_rtd_theme sphinxcontrib-napoleon

# Build the documentation
cd docs
make html  # On Windows: make.bat html

# View the documentation
# Open docs/_build/html/index.html in your browser

As a Library in Your Python Code

Basic Usage

from crawlit import Crawler, save_results, generate_summary_report

# Initialize the crawler with custom parameters
crawler = Crawler(
    start_url="https://example.com",
    max_depth=3,
    internal_only=True,
    user_agent="MyCustomBot/1.0",
    delay=0.5,
    respect_robots=True
)

# Start crawling
crawler.crawl()

# Get and process results
results = crawler.get_results()
print(f"Crawled {len(results)} URLs")

# Save results in different formats
save_results(results, "json", "crawl_results.json", pretty=True)

Advanced Usage with New Features

from crawlit import (
    Crawler, 
    SessionManager, 
    URLFilter, 
    ProgressTracker, 
    RateLimiter,
    PageCache,
    StorageManager,
    SitemapParser,
    ContentDeduplicator,
    LoggingConfig
)

# Configure enhanced logging
logging_config = LoggingConfig(
    level="INFO",
    json_logging=True,
    log_file="crawl.log",
    rotation_size_mb=10
)
logging_config.apply()

# Set up session with authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="your-oauth-token",
    api_key="your-api-key",
    api_key_header="X-API-Key"
)

# Configure URL filtering
url_filter = URLFilter(
    allowed_patterns=[r".*example\.com.*"],
    blocked_extensions=[".pdf", ".zip"]
)

# Set up progress tracking
def progress_callback(stats):
    print(f"Progress: {stats['crawled']}/{stats['total']} URLs")

progress_tracker = ProgressTracker(callback=progress_callback)

# Configure per-domain rate limiting
rate_limiter = RateLimiter(default_delay=0.5)
rate_limiter.set_domain_delay("example.com", 1.0)  # Slower for specific domain

# Set up page caching
page_cache = PageCache(use_disk=True, cache_dir="./cache", ttl=3600)

# Configure storage (optional HTML content storage)
storage_manager = StorageManager(
    store_html_content=True,
    use_disk_storage=True,
    storage_dir="./html_storage"
)

# Set up content deduplication
content_deduplicator = ContentDeduplicator(
    min_content_length=100,
    normalize_content=True
)

# Initialize crawler with all features
crawler = Crawler(
    start_url="https://example.com",
    max_depth=3,
    session_manager=session_manager,
    url_filter=url_filter,
    progress_tracker=progress_tracker,
    rate_limiter=rate_limiter,
    page_cache=page_cache,
    storage_manager=storage_manager,
    content_deduplicator=content_deduplicator,
    use_per_domain_delay=True,
    enable_content_extraction=True,
    enable_content_deduplication=True,
    use_sitemap=True,  # Auto-discover sitemaps from robots.txt
    max_workers=4,  # Multi-threaded crawling
    max_queue_size=1000
)

# Start crawling
crawler.crawl()

# Pause and resume
crawler.pause()
# ... do something ...
crawler.resume()

# Save state for later resumption
crawler.save_state("crawl_state.json")

# Get results
results = crawler.get_results()
print(f"Crawled {len(results)} URLs")

Asynchronous Usage

import asyncio
from crawlit import AsyncCrawler, AsyncRateLimiter

async def main():
    # Async crawler with async rate limiter
    rate_limiter = AsyncRateLimiter(default_delay=0.5)
    
    crawler = AsyncCrawler(
        start_url="https://example.com",
        max_depth=3,
        rate_limiter=rate_limiter,
        use_per_domain_delay=True,
        max_concurrent_requests=10
    )
    
    await crawler.crawl()
    results = crawler.get_results()
    print(f"Crawled {len(results)} URLs")

asyncio.run(main())

See the examples/programmatic_usage.py file for a complete example.

Command Line Interface

If you installed with pip install crawlit[cli], you can use the command-line interface:

# Basic usage
crawlit --url https://example.com

# Advanced options
crawlit --url https://example.com \
        --depth 3 \
        --output-format json \
        --output results.json \
        --delay 0.5 \
        --user-agent "crawlit/1.0" \
        --ignore-robots

# With new extraction features (v0.2.0+)
crawlit --url https://example.com \
        --user-agent "crawlit/2.0" \
        --extract-tables \
        --tables-output "./table_output" \
        --extract-images \
        --images-output "./image_output" \
        --extract-keywords \
        --keywords-output "keywords.json"

Command Line Arguments

Argument	Description	Default
--url, -u	Target website URL	Required
--depth, -d	Maximum crawl depth	3
--output-format, -f	Output format (json, csv, txt, html)	json
--output, -O	File to save results	crawl_results.json
--pretty-json, -p	Enable pretty-print JSON with indentation	False
--ignore-robots, -i	Ignore robots.txt rules	False
--delay	Delay between requests (seconds)	0.1
--user-agent, -a	Custom User-Agent string	crawlit/1.0
--allow-external, -e	Allow crawling URLs outside initial domain	False
--summary, -s	Show a summary of crawl results	False
--verbose, -v	Verbose output	False
--extract-keywords, -k	Extract keywords from crawled pages	False
--keywords-output	File to save extracted keywords	keywords.json
--max-keywords	Maximum number of keywords to extract per page	20
--min-word-length	Minimum length of words to consider as keywords	3
--extract-images, -img	Extract images from crawled pages	False
--images-output	Directory to save extracted images data	image_output/
--extract-tables, -t	Extract tables from crawled pages	False
--tables-output	Directory to save extracted tables	table_output/
--tables-format	Format to save extracted tables (csv or json)	csv
--min-rows	Minimum number of rows for a table to be extracted	1
--min-columns	Minimum number of columns for a table to be extracted	2
--max-table-depth	Maximum depth to extract tables from	Same as max crawl depth
--help, -h	Show help message	-

📊 Advanced Table Extraction

Crawlit includes powerful HTML table extraction capabilities:

from crawlit.extractors.tables import extract_tables

# Extract tables with minimum rows and columns filters
tables = extract_tables(html_content, min_rows=2, min_columns=3)

# Convert tables to CSV
from crawlit.extractors.tables import tables_to_csv
tables_to_csv(tables, base_filename="extracted_tables", output_dir="output")

# Convert to dictionaries using first row as headers
from crawlit.extractors.tables import tables_to_dict
table_dicts = tables_to_dict(tables)

# Convert to JSON
from crawlit.extractors.tables import tables_to_json
tables_to_json(tables, base_filename="extracted_tables")

The advanced table extraction provides:

• Smart handling of <thead> and <tbody> sections
• Full support for rowspan and colspan attributes
• Consistent column count across all rows
• Thorough cell content cleaning (HTML entities, whitespace, etc.)

For examples, see examples/enhanced_table_extraction.py and examples/rowspan_colspan_example.py.

🔍 Keyword Extraction

Crawlit 2.0 includes sophisticated keyword extraction capabilities:

from crawlit import Crawler
from crawlit.extractors.keyword_extractor import KeywordExtractor

# Option 1: Use the crawler with crawlit/2.0 user agent for automatic keyword extraction
crawler = Crawler(
    start_url="https://example.com",
    user_agent="crawlit/2.0",  # Required for keyword extraction
    max_depth=2
)
crawler.crawl()
results = crawler.get_results()

# Access extracted keywords from results
for url, data in results.items():
    if 'keywords' in data:
        print(f"Keywords for {url}: {data['keywords']}")
    if 'keyphrases' in data:
        print(f"Key phrases: {data['keyphrases']}")

# Option 2: Use the keyword extractor directly on HTML content
extractor = KeywordExtractor(min_word_length=4, max_keywords=10)
html_content = "<html><body><h1>Keyword Extraction Example</h1><p>This demonstrates advanced keyword extraction capability.</p></body></html>"

# Get keywords with scores
keywords_data = extractor.extract_keywords(html_content, include_scores=True)
print(f"Keywords: {keywords_data['keywords']}")
print(f"Scores: {keywords_data['scores']}")

# Get multi-word phrases
keyphrases = extractor.extract_keyphrases(html_content)
print(f"Key phrases: {keyphrases}")

The keyword extraction offers:

• Smart weighting of content based on HTML structure (headings, titles, etc.)
• Automatic filtering of common stop words
• Multi-word phrase extraction for more context-rich keywords
• Scoring based on frequency and relevance
• Integration with crawl results

For a complete example, see examples/keyword_extraction.py.

🖼️ Image Extraction

Crawlit v0.2.0 introduces comprehensive image extraction capabilities:

from crawlit import Crawler
from crawlit.extractors.image_extractor import ImageTagParser

# Option 1: Using the crawler for automatic image extraction
crawler = Crawler(
    start_url="https://example.com",
    max_depth=2
)
crawler.crawl()
results = crawler.get_results()

# Access extracted images from results
for url, data in results.items():
    if 'images' in data:
        for img in data['images']:
            print(f"Image URL: {img.get('src')}")
            print(f"Alt text: {img.get('alt', 'None')}")
            if 'width' in img and 'height' in img:
                print(f"Dimensions: {img['width']}x{img['height']}")
            print(f"Decorative: {img.get('decorative', False)}")

# Option 2: Use the image extractor directly on HTML content
from html.parser import HTMLParser
parser = ImageTagParser()
html_content = "<html><body><img src='example.jpg' alt='Example'></body></html>"
parser.feed(html_content)
images = parser.images

The image extraction feature provides:

• Complete metadata extraction (src, alt, width, height, etc.)
• Parent element context to understand image placement
• Accessibility analysis (identifying decorative images missing alt text)
• Integration with crawl results

For a complete example, see examples/image_extraction.py.

🔐 Authentication & Session Management

Crawlit supports multiple authentication methods:

from crawlit import Crawler, SessionManager

# OAuth token authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="your-oauth-token"
)

# API key authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    api_key="your-api-key",
    api_key_header="X-API-Key"  # Optional, defaults to "Authorization"
)

# Basic authentication
from requests.auth import HTTPBasicAuth
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    auth=HTTPBasicAuth("username", "password")
)

# Custom headers
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    headers={"X-Custom-Header": "value"}
)

# Combine multiple methods
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="token",
    api_key="key",
    headers={"X-Custom": "value"}
)

# Use with crawler
crawler = Crawler(
    start_url="https://example.com",
    session_manager=session_manager
)

⚡ Rate Limiting

Per-Domain Rate Limiting

from crawlit import Crawler, RateLimiter

# Create rate limiter with default delay
rate_limiter = RateLimiter(default_delay=0.5)

# Set custom delays for specific domains
rate_limiter.set_domain_delay("example.com", 1.0)
rate_limiter.set_domain_delay("api.example.com", 2.0)

# Use with crawler (automatically respects robots.txt crawl-delay)
crawler = Crawler(
    start_url="https://example.com",
    rate_limiter=rate_limiter,
    use_per_domain_delay=True
)

The crawler automatically extracts and respects Crawl-delay directives from robots.txt files.

🗺️ Sitemap Support

from crawlit import Crawler

# Auto-discover sitemaps from robots.txt
crawler = Crawler(
    start_url="https://example.com",
    use_sitemap=True  # Automatically discovers and parses sitemaps
)

# Or provide explicit sitemap URLs
crawler = Crawler(
    start_url="https://example.com",
    use_sitemap=True,
    sitemap_urls=[
        "https://example.com/sitemap.xml",
        "https://example.com/sitemap-news.xml"
    ]
)

📊 Progress Tracking

from crawlit import Crawler, ProgressTracker

def progress_callback(stats):
    print(f"Crawled: {stats['crawled']}, Failed: {stats['failed']}, Total: {stats['total']}")
    print(f"Success Rate: {stats['success_rate']:.2f}%")

progress_tracker = ProgressTracker(callback=progress_callback)

crawler = Crawler(
    start_url="https://example.com",
    progress_tracker=progress_tracker
)

crawler.crawl()

# Get final statistics
stats = progress_tracker.get_stats()
print(f"Final stats: {stats}")

🔍 URL Filtering

from crawlit import Crawler, URLFilter

# Filter by patterns
url_filter = URLFilter(
    allowed_patterns=[r".*example\.com.*"],
    blocked_patterns=[r".*admin.*"]
)

# Filter by file extensions
url_filter = URLFilter(
    allowed_extensions=[".html", ".htm"],
    blocked_extensions=[".pdf", ".zip"]
)

# Use factory methods
url_filter = URLFilter.html_only()  # Only HTML pages
url_filter = URLFilter.exclude_media()  # Exclude images, videos, etc.

# Use with crawler
crawler = Crawler(
    start_url="https://example.com",
    url_filter=url_filter
)

💾 Caching & Resume

from crawlit import Crawler, PageCache, CrawlResume

# Set up page caching
page_cache = PageCache(
    use_disk=True,
    cache_dir="./cache",
    ttl=3600  # Cache for 1 hour
)

crawler = Crawler(
    start_url="https://example.com",
    page_cache=page_cache
)

# Save state for resumption
crawler.save_state("crawl_state.json")

# Later, resume from saved state
crawler.load_state("crawl_state.json")
crawler.crawl()  # Continues from where it left off

🗄️ Storage Management

from crawlit import Crawler, StorageManager

# Store HTML content in memory (default)
storage_manager = StorageManager(store_html_content=True)

# Disable HTML content storage (save memory)
storage_manager = StorageManager(store_html_content=False)

# Store HTML content on disk
storage_manager = StorageManager(
    store_html_content=True,
    use_disk_storage=True,
    storage_dir="./html_storage"
)

crawler = Crawler(
    start_url="https://example.com",
    storage_manager=storage_manager
)

🔄 Content Deduplication

from crawlit import Crawler, ContentDeduplicator

# Set up content deduplication
content_deduplicator = ContentDeduplicator(
    min_content_length=100,  # Minimum content length to consider
    normalize_content=True   # Normalize content (remove whitespace, etc.)
)

crawler = Crawler(
    start_url="https://example.com",
    content_deduplicator=content_deduplicator,
    enable_content_deduplication=True
)

# Get deduplication statistics
stats = content_deduplicator.get_stats()
print(f"Duplicate pages found: {stats['duplicates']}")

📝 Enhanced Logging

from crawlit import LoggingConfig, configure_logging

# Configure structured JSON logging
logging_config = LoggingConfig(
    level="INFO",
    json_logging=True,
    log_file="crawl.log",
    rotation_size_mb=10,  # Rotate when file reaches 10MB
    rotation_time="midnight"  # Or rotate daily at midnight
)

logging_config.apply()

# Or use convenience function
configure_logging(
    level="INFO",
    log_file="crawl.log",
    json_logging=True
)

# Use contextual logging
from crawlit import log_with_context

log_with_context("INFO", "Starting crawl", {"url": "https://example.com"})

🧵 Multi-threaded Crawling

from crawlit import Crawler

# Enable multi-threaded crawling (sync crawler only)
crawler = Crawler(
    start_url="https://example.com",
    max_workers=4  # Use 4 worker threads
)

# Single-threaded (default)
crawler = Crawler(
    start_url="https://example.com",
    max_workers=1  # Or omit for default
)

⏸️ Pause & Resume

from crawlit import Crawler

crawler = Crawler(start_url="https://example.com")

# Start crawling in background thread
import threading
thread = threading.Thread(target=crawler.crawl)
thread.start()

# Pause crawling
crawler.pause()

# Check if paused
if crawler.is_paused():
    print("Crawler is paused")

# Resume crawling
crawler.resume()

# Get queue statistics
stats = crawler.get_queue_stats()
print(f"Queue size: {stats['size']}")

🏗️ Project Structure

crawlit/
├── crawlit.py           # CLI entry point
├── requirements.txt     # Project dependencies
├── crawler/             # Core crawler modules
│   ├── __init__.py
│   ├── engine.py        # Synchronous crawler engine
│   ├── async_engine.py  # Asynchronous crawler engine
│   ├── fetcher.py       # HTTP request handling (sync)
│   ├── async_fetcher.py # HTTP request handling (async)
│   ├── parser.py        # HTML parsing and link extraction
│   └── robots.py        # Robots.txt parser with crawl-delay support
├── extractors/          # Data extraction modules
│   ├── __init__.py
│   ├── content_extractor.py  # Unified content extraction
│   ├── image_extractor.py    # Image extraction and analysis
│   ├── keyword_extractor.py  # Keyword and keyphrase extraction
│   └── tables.py            # Advanced table extraction
├── utils/               # Utility modules
│   ├── __init__.py
│   ├── session_manager.py   # Session management with auth
│   ├── url_filter.py         # Advanced URL filtering
│   ├── progress.py           # Progress tracking
│   ├── queue_manager.py      # Queue state management
│   ├── cache.py              # Page caching and resume
│   ├── storage.py            # HTML content storage
│   ├── sitemap.py            # Sitemap parsing
│   ├── rate_limiter.py       # Per-domain rate limiting
│   ├── deduplication.py      # Content deduplication
│   ├── logging_config.py     # Enhanced logging
│   └── errors.py             # Custom exceptions
├── output/              # Output formatters
│   ├── __init__.py
│   └── formatters.py    # Output formatting functions
├── examples/            # Example usage
│   ├── programmatic_usage.py  # Example of using as a library
│   ├── authenticated_crawling.py  # Authentication examples
│   └── extraction_coverage_demo.py  # Extraction features demo
└── tests/               # Unit and integration tests
    └── __init__.py

📅 Project Timeline

• May 2025: Initial structure and CLI setup
• May 15, 2025: v0.2.0 release with image extraction, table extraction, and keyword extraction features
• June 2025: Core functionality complete (HTTP handling, parsing, domain control)
• June 30, 2025: Project completion target with all core features

🤝 Contributing

Contributions will be welcome after the core functionality is complete. Please check back after June 30, 2025, for contribution guidelines.

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Built and maintained by Swayam Dani

---

Note: This project is under active development with completion targeted for June 30, 2025.