Supacrawl
Clean markdown from the web, for the terminal and for AI assistants.
One pip install, no Docker, no database, no hosted account: a CLI, a Python library, a Firecrawl-compatible REST API and an MCP server over the same engine.
Installation · Features · Benchmark · Docs
The Problem
A model that reads the web needs the article, not the navigation, the cookie banner and the footer around it. Pages that need JavaScript, sit behind a bot manager, or are PDFs break a plain HTTP fetch, and the tools that handle them need a server, a queue and a database, or a paid key.
The Solution
Supacrawl fetches the cheap way first and escalates only when a page needs it: an HTTP GET, then a headless browser, then stealth engines, with a quality verdict on every result so the caller knows what it got. It remembers per domain what worked. Measured on 01/11/2026 against the open-source field over 33 public pages, three pages at a time, on one 20-core host that was busy throughout (load average 29 to 65), so the times compare the readers with each other and are not a speed claim for any of them (method and full table):
| Reader | Pages | Content kept | Boilerplate gone | Median s | Peak MB |
|---|---|---|---|---|---|
| supacrawl 2026.11.0 | 88% | 93% | 92% | 9.7 | 2,642 |
| Firecrawl self-hosted | 85% | 95% | 40% | 3.6 | 77 (client only) |
| Crawl4AI 0.9.5 | 82% | 96% | 56% | 5.2 | 2,667 (overstated) |
| Jina Reader (keyless) | 94% | 95% | 24% | 2.1 | 83 |
| trafilatura 2.3.1 | 76% | 74% | 97% | 0.8 | 174 |
Supacrawl is the one reader that both keeps the content and drops the boilerplate, and the one with web search; it pays in time and memory where a page forces the escalation ladder. One page at a time, on the same host, its median was less than half the concurrent figure and its peak memory about half. Peak MB is the process tree's proportional set size; the benchmark page says how each figure was taken.
Features
| HTTP first, browser when needed A static page never starts a browser; JavaScript, bot challenges and PDFs escalate automatically |
Honest quality signal Every result says ok, thin, js_shell, bot_challenge, captcha or paywall, with the next step and every strategy that ran |
| Three-tier anti-bot ladder Playwright, Patchright, Camoufox, climbed on a poor verdict and remembered per domain |
Retrieval for long pagesquery keeps the sections that match, section one heading's part; a section the query names comes first |
| Structured data without an LLM schema.org JSON-LD, microdata and OpenGraph as JSON, deterministically; LLM extraction and summaries through Ollama, OpenAI, Anthropic or any OpenAI-compatible gateway |
Search, crawl, map, PDFs Multi-provider web search with recency and domain filters, robots-aware crawling to a folder, URL discovery, PDF text with optional OCR |
Installation
Prerequisites: Python 3.14+
pip install supacrawl
playwright install chromium
supacrawl scrape https://example.com
Add supacrawl[stealth] and supacrawl[camoufox] (then patchright install chromium and camoufox fetch) for the full anti-bot ladder, supacrawl[api] for the REST server, supacrawl[pdf-ocr] for scanned PDFs. Set BRAVE_API_KEY for search. See Configuration for every option.
Documentation
| Guide | Description |
|---|---|
| CLI reference | Every command and option |
| Configuration | Settings, environment variables, LLM providers, search providers, telemetry |
| MCP server | Tools, transports, serving an untrusted caller, and what the server is built on |
| REST API | The Firecrawl v2-compatible endpoints and the n8n integration |
| Benchmark | How the figures above were measured, and how to re-run them |
| Agent skill | Onboarding for an agent that drives the CLI; supacrawl install-skill registers it |
| Product definition | Intent, competitive reference, decisions |
Contributing
git clone https://github.com/radar-hooves/supacrawl.git
cd supacrawl
uv sync --all-groups --all-extras && playwright install chromium
ruff check src tests && mypy src && pytest -q
Tests that drive a real browser are marked browser; see docs/development/testing.md.
1 maintainer
Metadata
1 maintainer
Release files for supacrawl 2026.11.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| supacrawl-2026.11.3.tar.gz | 607.0 kB | Without provenance | Details for supacrawl-2026.11.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| supacrawl-2026.11.3-py3-none-any.whl | Python 3 | none | any | Details for supacrawl-2026.11.3-py3-none-any.whl |
Total release size: 1.0 MB
Release files / supacrawl-2026.11.3.tar.gz
| Download URL | supacrawl-2026.11.3.tar.gz |
|---|---|
| Size | 607.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a722d8d6d8660e3cde6a2abdfb60c3030bc4fc0c65dcb51a9c2abf2186459c2
|
|
BLAKE2b-256 checksum How to use checksums |
5282c1c0362e9113690298b9aeb0b587e91aa143b8d88df19b40c24522a8fb1e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"NixOS","version":"26.05","id":"yarara","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / supacrawl-2026.11.3-py3-none-any.whl
| Download URL | supacrawl-2026.11.3-py3-none-any.whl |
|---|---|
| Size | 434.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bf277bd575287ff5faf82222f801f057ad10c2c06b8199bf81241b9c7ed3e48e
|
|
BLAKE2b-256 checksum How to use checksums |
080b3e4b8b6df5da482f4f9a6fe9c2c9dfeb7a0eb60a40b1abed3ff442d9e015
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"NixOS","version":"26.05","id":"yarara","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for supacrawl 2026.11.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
