The Problem

A model that reads the web needs the article, not the navigation, the cookie banner and the footer around it. Pages that need JavaScript, sit behind a bot manager, or are PDFs break a plain HTTP fetch, and the tools that handle them need a server, a queue and a database, or a paid key.

The Solution

Supacrawl fetches the cheap way first and escalates only when a page needs it: an HTTP GET, then a headless browser, then stealth engines, with a quality verdict on every result so the caller knows what it got. It remembers per domain what worked. Measured on 01/11/2026 against the open-source field over 33 public pages, three pages at a time, on one 20-core host that was busy throughout (load average 29 to 65), so the times compare the readers with each other and are not a speed claim for any of them (method and full table):

Supacrawl is the one reader that both keeps the content and drops the boilerplate, and the one with web search; it pays in time and memory where a page forces the escalation ladder. One page at a time, on the same host, its median was less than half the concurrent figure and its peak memory about half. Peak MB is the process tree's proportional set size; the benchmark page says how each figure was taken.

Features

Installation

Prerequisites: Python 3.14+

pip install supacrawl
playwright install chromium
supacrawl scrape https://example.com

Add supacrawl[stealth] and supacrawl[camoufox] (then patchright install chromium and camoufox fetch) for the full anti-bot ladder, supacrawl[api] for the REST server, supacrawl[pdf-ocr] for scanned PDFs. Set BRAVE_API_KEY for search. See Configuration for every option.

Documentation

Contributing

git clone https://github.com/radar-hooves/supacrawl.git
cd supacrawl
uv sync --all-groups --all-extras && playwright install chromium
ruff check src tests && mypy src && pytest -q

Tests that drive a real browser are marked browser; see docs/development/testing.md.