wscrape

Fast async web scraper — CLI tool, Python library, and AI agent skill.

wscrape is a production-ready Python package that scrapes websites, optionally crawls links recursively, and returns structured data suitable for AI consumption.

Installation

pip install wscrape

For development:

pip install wscrape[dev]

CLI Usage

# Scrape a single page
wscrape https://example.com

# Deep crawl with depth limit
wscrape https://example.com --deep --max-depth 2

# Save results to a file
wscrape https://example.com --deep --save --output data.json

# CSV output
wscrape https://example.com --format csv --save --output data.csv

# Custom rate limiting (milliseconds between requests)
wscrape https://example.com --deep --rate 500

CLI Options

Python Library Usage

from wscrape import run_scraper

# Synchronous — works everywhere
results = run_scraper("https://example.com", deep=True, max_depth=2)
for page in results:
    print(page["url"], page["title"])

Async Usage

import asyncio
from wscrape import async_run_scraper

results = asyncio.run(
    async_run_scraper("https://example.com", deep=True, max_depth=2)
)

AI Agent Skill

wscrape is designed to be called programmatically by AI agents (Claude, OpenAI, etc.):

from wscrape import run_scraper

# Returns list[dict] with keys: url, title, text, links, depth
data = run_scraper("https://example.com")

Output Format

Each scraped page produces:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples ...",
  "links": ["https://www.iana.org/domains/example"],
  "depth": 0
}

Architecture

wscrape/
  __init__.py      # Public API & AI skill interface (run_scraper, async_run_scraper)
  __main__.py      # python -m wscrape support
  cli.py           # CLI entry point (argparse)
  config.py        # ScrapeConfig dataclass
  exceptions.py    # Custom exception hierarchy
  core/
    fetcher.py     # Async HTTP client with retries & backoff
    parser.py      # HTML parsing (lxml) — title, text, links, CSS, XPath
    crawler.py     # BFS crawler with rate limiting & same-domain policy
    output.py      # JSON / CSV serialisation & file output
tests/
  test_fetcher.py  # Fetcher unit tests (mocked HTTP)
  test_parser.py   # Parser unit tests
  test_crawler.py  # Crawler unit tests (mocked HTTP)
  test_cli.py      # CLI integration tests

Development

git clone https://github.com/wscrape/wscrape.git
cd wscrape
pip install -e ".[dev]"
pytest

License

MIT