Trace Scraper

High-performance async web scraper for Python. Scrape JavaScript-rendered pages, crawl websites recursively, parse sitemaps, and extract clean content — all with built-in anti-bot bypass.

Why Trace Scraper?

• Hybrid Engine: Uses httpx for fast static pages, Playwright for JavaScript-heavy sites
• Anti-Bot Bypass: Built-in detection evasion with realistic browser fingerprints
• Async & Parallel: Scrape hundreds of pages concurrently with asyncio
• Multiple Modes: Single page, recursive crawl, sitemap parsing, URL lists
• Clean Output: Automatic HTML cleanup, content deduplication, PDF extraction
• Type Safe: Full type hints with py.typed marker

Installation

pip install trace-scraper

Install Playwright browsers (required for JS rendering):

playwright install chromium

Quick Start

Command Line

# Scrape a single page
trace https://example.com

# Crawl entire website
trace https://docs.example.com --mode recursive --max-pages 100

# Parse sitemap
trace https://example.com/sitemap.xml --mode sitemap

# Use browser for JavaScript sites
trace https://spa-website.com --browser

# Save to JSON
trace https://example.com -o output.json

Python API

import asyncio
from trace_scraper import WebScraper, ScrapeMode

async def main():
    async with WebScraper() as scraper:
        async for doc in scraper.scrape("https://example.com", mode=ScrapeMode.SINGLE):
            print(f"Title: {doc.semantic_identifier}")
            print(f"Content: {doc.sections[0].text}")

asyncio.run(main())

Advanced Configuration

from trace_scraper import WebScraper, ScraperConfig, ScrapeMode

config = ScraperConfig(
    max_concurrent_pages=20,     # Parallel scraping
    page_timeout=15.0,           # Timeout per page
    scroll_pages=True,           # Load lazy content
    extract_pdfs=True,           # Extract PDF text
    use_browser_for_all=False,   # Use httpx when possible
)

async with WebScraper(config) as scraper:
    async for doc in scraper.scrape(
        "https://docs.example.com",
        mode=ScrapeMode.RECURSIVE
    ):
        print(doc.semantic_identifier)

CLI Options

Configuration Options

Features

Scraping Modes

• Single: Scrape one URL
• Recursive: Crawl entire site following internal links
• Sitemap: Parse sitemap.xml and scrape all URLs
• URL List: Scrape a provided list of URLs

Content Extraction

• Automatic removal of scripts, styles, navigation, footers
• Clean text extraction with structure preservation
• Link extraction with markdown formatting
• PDF text extraction (optional dependency)

Anti-Bot Features

• Realistic browser fingerprints
• Custom User-Agent and headers
• Stealth mode scripts
• Automatic retry with browser fallback

Development

# Clone repository
git clone https://github.com/boxed-dev/trace.git
cd trace

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linter
ruff check .

# Type check
mypy trace_scraper

License

MIT License - see LICENSE for details.