scrape

Fast async web scraper — CLI tool, Python library, and AI agent skill.

scrape is a production-ready Python package that scrapes websites, optionally crawls links recursively, and returns structured data suitable for AI consumption.

Installation

pip install scrape

For development:

pip install scrape[dev]

CLI Usage

# Scrape a single page
scrape https://example.com

# Deep crawl with depth limit
scrape https://example.com --deep --max-depth 2

# Save results to a file
scrape https://example.com --deep --save --output data.json

# CSV output
scrape https://example.com --format csv --save --output data.csv

# JSON Lines output (streaming-friendly)
scrape https://example.com --format jsonl --save --output data.jsonl

# Custom rate limiting and concurrency
scrape https://example.com --deep --rate 500 --concurrency 3

# Limit max pages
scrape https://example.com --deep --max-pages 50

# Allow external domain crawling
scrape https://example.com --deep --allow-external

# Use a proxy
scrape https://example.com --proxy http://proxy:8080

# Quiet mode (errors only)
scrape https://example.com --quiet

CLI Options

Python Library Usage

from scrape import run_scraper

# Synchronous — works everywhere
results = run_scraper("https://example.com", deep=True, max_depth=2)
for page in results:
    print(page["url"], page["title"])

Async Usage

import asyncio
from scrape import async_run_scraper

results = asyncio.run(
    async_run_scraper("https://example.com", deep=True, max_depth=2)
)

Batch Scraping

from scrape import scrape_urls

# Scrape multiple URLs concurrently
results = scrape_urls(["https://example.com", "https://example.org"])

AI Agent Skill

scrape is designed to be called programmatically by AI agents (Claude, OpenAI, etc.):

from scrape import run_scraper

# Returns list[dict] with keys: url, title, text, links, images, meta_description, meta_author, depth
data = run_scraper("https://example.com")

Output Format

Each scraped page produces:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples ...",
  "links": ["https://www.iana.org/domains/example"],
  "images": [],
  "meta_description": "",
  "meta_author": "",
  "depth": 0
}

Architecture

scrape/
  __init__.py      # Public API & AI skill interface (run_scraper, async_run_scraper, scrape_urls)
  __main__.py      # python -m scrape support
  cli.py           # CLI entry point (argparse)
  config.py        # ScrapeConfig dataclass
  exceptions.py    # Custom exception hierarchy
  core/
    fetcher.py     # Async HTTP client with retries, backoff, jitter, proxy support
    parser.py      # HTML parsing (lxml) — title, text, links, images, metadata, CSS, XPath
    crawler.py     # BFS crawler with concurrency, rate limiting & domain policy
    output.py      # JSON / CSV / JSONL serialisation & file output
tests/
  test_fetcher.py  # Fetcher unit tests (mocked HTTP)
  test_parser.py   # Parser unit tests
  test_crawler.py  # Crawler unit tests (mocked HTTP)
  test_cli.py      # CLI integration tests

Development

git clone https://github.com/wscrape/scrape.git
cd scrape
pip install -e ".[dev]"
pytest

License

MIT