scrapefold 0.10.0

Turn any URL into clean markdown.
One async Python interface over 38 URL engines — with automatic anti-bot escalation and LLM-ready output. Search delegates to Enrichfold.

38 URL engines · 5 anti-bot stacks handled (Cloudflare · Datadome · PerimeterX · Akamai · AWS WAF) · MIT

⭐ If Scrapefold saves you a vendor rewrite, star the repo — it's the #1 way to help others find it.

Scrapefold is the open-source scraping engine from Datatera.ai — extracted from our commercial enterprise AI data platform and battle-tested in production against Cloudflare, Datadome, PerimeterX, and Akamai-protected sites.

30-second taste

import asyncio
from scrapefold import scrape

async def main():
    result = await scrape("https://example.com")   # router picks the cheapest engine that works
    print(result.markdown)                          # always populated — LLM-ready
    print(result.engine, result.elapsed_ms)         # which engine ran, and how fast

asyncio.run(main())

pip install scrapefold

Need a stealth browser, a paid vendor, or a whole-site crawl? Same call — Scrapefold escalates only as far as it has to. See the full quickstart →

Engine Comparison

Each row reflects the engine's typical behaviour against the four hardest target classes: static HTML, JS-rendered SPA, Cloudflare/Datadome-walled, and IP-geofenced. Run your own: scrapefold list-engines then scrapefold scrape <url> --engines <name>.

★★★ Excellent ★★☆ Good ★☆☆ Basic ☆☆☆ Not supported — $ ~$0.1–0.5/1K req $$ ~$1–3/1K req $$$ ~$5–15/1K req

Full ladder, site-class routing, and budget enforcement →

search() delegates to Enrichfold, which owns search adapters and result fusion. Scrapefold retains the existing API for callers that also fetch pages. Search adapters are enabled when their API key is present: EXA_API_KEY, PARALLEL_API_KEY, YOU_API_KEY, TAVILY_API_KEY, LINKUP_API_KEY, SELTZ_API_KEY, TINYFISH_API_KEY, NIMBLE_API_KEY, or BROWSERBASE_API_KEY. Use SearchOptions(engines=("parallel", "tavily")) to choose a fixed set. URL fetch uses ScrapeOptions(engines=("nimble",)), ("tinyfish",), or ("linkup",) with the same vendor key. Browserbase Fetch works with ScrapeOptions(engines=("browserbase",)).

From the pictured services, Exa, Parallel, You.com, Tavily, Seltz, TinyFish, Nimble, Linkup, and Browserbase belong to Enrichfold search. Scrapefold fetches URLs with Exa, Oxylabs, Firecrawl, TinyFish, Nimble, Linkup, and Browserbase. Kernel provides browser sessions and Browser Use runs browser tasks; these need a separate browser workflow to be useful.

How to Choose

Why Scrapefold?

Every scraping vendor has trade-offs. Scrapefold lets you switch between them with one line:

import asyncio
from scrapefold import scrape, crawl_site, extract, ScrapeOptions

async def main():
    # Single URL, auto-engine — router picks the cheapest tier that works
    result = await scrape("https://example.com")
    print(result.markdown)        # always populated
    print(result.engine)          # which engine actually fetched it
    print(result.elapsed_ms)

    # Cloudflare-protected site — same call, router auto-escalates
    result = await scrape(
        "https://protected.example.com",
        opts=ScrapeOptions(render_js=True, stealth=True),
    )

    # Whole-site crawl with disk cache
    crawl = await crawl_site(
        "https://docs.example.com",
        # autothrottle: adaptive per-host delay — polite on slow / rate-limiting origins
        opts=ScrapeOptions(max_pages=50, max_depth=3, autothrottle=True),
        output="site.md",                       # stitched markdown
        per_page_dir="pages/",                  # one .md per URL
        cache_dir="~/.scrapefold/cache",
        cache_ttl_days=7,
    )
    print(f"{len(crawl.pages)} pages, {len(crawl.failures)} failures")

    # Structured extraction via YOUR LLM — scrapefold ships no vendor LLM SDK
    async def my_llm(prompt: str) -> str:
        ...  # call Claude / GPT / a local model — anything that returns text

    data = await extract(
        await scrape("https://shop.example.com/widget"),
        schema={"type": "object", "required": ["title", "price"]},  # or plain English
        llm=my_llm,
    )
    print(data)                   # {"title": ..., "price": ...}

asyncio.run(main())

Supported Engines

Adding your own engine? Implement the ScrapeEngine interface — see Adding a Custom Engine below and CONTRIBUTING.md for the 5-step checklist.

Installation

# Core only — requests engine, no third-party deps
pip install scrapefold

# One specific vendor
pip install "scrapefold[firecrawl]"
pip install "scrapefold[scrapling,jina]"
pip install "scrapefold[pixelrag]"  # PixelRAG upstream requires Python 3.12+

# Everything
pip install "scrapefold[all]"

# MCP server for AI agents (Claude Code, Cursor, etc.)
pip install "scrapefold[mcp]"

Requires Python 3.10+.

PixelRAG's upstream package currently requires Python 3.12+. On Python 3.10/3.11, run pixelshot from a separate environment and point PIXELRAG_BIN at that executable.

CLI

# Single URL → markdown
scrapefold scrape https://example.com

# Pick a specific engine
scrapefold scrape https://example.com --engines firecrawl --json

# Whole-site crawl
scrapefold crawl https://docs.example.com --max-pages 50 --output site.md

# One .md per URL (for downstream parsers)
scrapefold crawl https://docs.example.com --per-page-dir pages/

# List engines and their availability
scrapefold list-engines

# Classify a URL's site class (cloudflare_protected / datadome_protected / etc.)
scrapefold classify https://example.com

# One-click MCP registration into your AI client (see below)
scrapefold install claude

# Health check: version, MCP extra, per-engine availability
scrapefold doctor

# Self-update (or just check): pip upgrade via the running interpreter
scrapefold update --check
scrapefold update --extras mcp

MCP Server (for Claude Code, Cursor, agents)

One-click install:

pip install "scrapefold[mcp]"
scrapefold install claude    # Claude Code
scrapefold install codex     # Codex CLI
scrapefold install cursor    # Cursor
scrapefold install vscode    # VS Code
scrapefold install generic   # print JSON config for any other client

Or drop into any client's MCP config manually:

{ "mcpServers": { "scrapefold": { "command": "scrapefold-mcp", "args": [] } } }

Exposes scrape_url, crawl_site, list_engines, classify_url tools over stdio — all 4 tool definitions + instructions cost your agent ≈750 tokens (budget-tested). Failures come back structured ({"error", "failures"}), never as error-page HTML posing as content.

Setting up with an AI agent? Just tell it:

fetch https://scrapefold.com/install.md

— a complete, agent-readable setup instruction (install → verify CLI → register MCP). It's idempotent: safe to run on a machine where scrapefold is already set up.

How is this different from Firecrawl / Crawl4AI / Scrapling / Jina?

They're not competitors — they're engines inside scrapefold. Firecrawl, Crawl4AI, Scrapling, Jina Reader, ScrapingBee, and 20+ others are all wrapped behind one ScrapeOptions/ScrapeResult interface. scrapefold's job is the layer above any single vendor:

• Routing — classify the URL (Cloudflare-protected? LinkedIn? plain blog?) and walk a per-class ladder from free local engines to paid APIs, stopping at the first good response.
• Honesty — block pages and CAPTCHA shells are detected as suspicious and trigger escalation instead of being returned as "content".
• One interface — switching vendors is a one-string change, not a pipeline rewrite. The MCP server stays deliberately small: 4 tight tools, no bloat in your agent's context.

If you love one vendor, pin it: --engines firecrawl. If you don't want to choose, don't: scrape(url) picks the cheapest tier that works.

Unified Result Format

Every engine returns the same ScrapeResult dataclass:

@dataclass(frozen=True, slots=True)
class ScrapeResult:
    url: str                   # final URL after redirects
    text: str                  # plain text — always populated
    markdown: str              # markdown — always populated
    html: str | None           # raw HTML when the engine returned it
    json: dict | None          # structured data when native
    engine: str                # which engine produced this
    elapsed_ms: int            # wall-clock time
    meta: dict                 # engine-specific metadata (status_code, headers, ...)

And crawl_site() returns:

@dataclass(frozen=True, slots=True)
class CrawlResult:
    pages: tuple[ScrapeResult, ...]
    stitched_path: Path              # all pages concatenated to one .md
    failures: tuple[str, ...]        # "<url>:<ExceptionType>:<detail>"

Anti-bot Detection

Scrapefold ships a content-quality detection module (scrapefold.detection) that decides when the router should escalate to a more expensive engine:

from scrapefold.detection import is_suspicious, reclassify_from_response

# is_suspicious returns True on:
# - empty / whitespace-only response
# - short text + HTTP 4xx/5xx
# - antibot phrases ("Just a moment...", "Verify you are human", ...)
# - >50% <noscript> domination
# - >90% <script> domination
# - HTTP 403 / 429 / 503 regardless of body length

# reclassify_from_response detects vendor anti-bot stacks from cookies/headers:
# Cloudflare, Datadome, PerimeterX, Akamai
site_class = reclassify_from_response(
    body=response.text,
    cookies=response.cookies,
    headers=response.headers,
    status_code=response.status_code,
)
# → "cloudflare_protected" | "datadome_protected" | None

Architecture

                        ┌──────────────────────────────┐
                        │      Your Application        │
                        └──────────┬───────────────────┘
                                   │
                        ┌──────────▼───────────────────┐
                        │       ScrapeRouter           │
                        │   scrape() / crawl_site()    │
                        └──────────┬───────────────────┘
                                   │
       ┌──────────┬───────┬────────┴────────┬──────────┬──────────┐
       ▼          ▼       ▼                 ▼          ▼          ▼
  ┌──────────┐ ┌──────────┐ ┌────────────┐ ┌──────────┐ ┌──────────┐
  │ requests │ │ scrapling│ │  crawl4ai  │ │ cloak    │ │ selenium │
  │  (local) │ │ stealth  │ │  (local)   │ │ browser  │ │ (local)  │
  └──────────┘ └──────────┘ └────────────┘ └──────────┘ └──────────┘
       │             │             │             │            │
  ┌──────────┐ ┌──────────┐ ┌────────────┐ ┌──────────┐ ┌──────────┐
  │ Jina     │ │ Firecrawl│ │ Scraping   │ │ Scraping │ │ Cloudfl. │
  │ Reader   │ │ (SaaS)   │ │ Bee (SaaS) │ │ dog SaaS │ │ BR SaaS  │
  └──────────┘ └──────────┘ └────────────┘ └──────────┘ └──────────┘
       │             │             │             │            │
       └─────────────┴──────┬──────┴─────────────┴────────────┘
                            │
                  ┌─────────▼────────┐
                  │   ScrapeResult   │
                  │  (text/markdown/ │
                  │   html/json)     │
                  └──────────────────┘

Engine Selection Logic

When no engine is explicitly specified, the router selects one automatically:

1. Explicit pin — ScrapeOptions(engines=("firecrawl",)) overrides everything.
2. Site class — classifier inspects URL + cookies/headers; e.g., a Cloudflare-protected site routes to the cloudflare_protected ladder.
3. Capability filter — ScrapeOptions(render_js=True, stealth=True) drops engines whose EngineCapabilities don't support those features.
4. Cost-ordered cascade — within the eligible set, try cheapest first; escalate on is_suspicious or AllEnginesFailed.

# Pin to specific engines (order matters)
opts = ScrapeOptions(engines=("requests", "scrapling_stealth", "firecrawl"))

# Restrict by capability — router picks the cheapest available
opts = ScrapeOptions(render_js=True, stealth=True)

# CLI equivalent
# scrapefold scrape <url> --engines requests,scrapling_stealth,firecrawl

Adding a Custom Engine

Implement the ScrapeEngine interface:

from scrapefold.engines.base import ScrapeEngine, EngineCapabilities
from scrapefold.options import ScrapeOptions
from scrapefold.result import ScrapeResult

class MyEngine(ScrapeEngine):
    NAME = "my_engine"
    CAPABILITIES = EngineCapabilities(
        supports_js=True,
        supports_stealth=False,
        avg_response_mb_estimate=2.0,
        cost_per_1k_requests_usd=1.50,
    )
    SUPPORTED_OPTIONS = {"render_js", "language", "headers"}

    def is_available(self) -> bool:
        try:
            import my_library  # noqa: F401
            return True
        except ImportError:
            return False

    async def _fetch(self, url: str, opts: ScrapeOptions) -> ScrapeResult:
        html = await my_library.fetch(url)
        return ScrapeResult(
            url=url,
            text=html_to_text(html),
            markdown=html_to_markdown(html),
            html=html,
            engine=self.NAME,
            elapsed_ms=0,  # populated by the base class
        )

# Register it
from scrapefold.engines.base import register
register("my_engine", MyEngine)

Full 5-step checklist: CONTRIBUTING.md.

Related Projects

Scrapefold integrates with these excellent projects:

Built by

Scrapefold is built and maintained by Mike Sadofyev (CEO, Datatera.ai), alongside a small ecosystem of AI-data tooling:

Connect: LinkedIn · X / Twitter · GitHub

⭐ Found Scrapefold useful? Star it on GitHub and share it — it genuinely helps the project reach more developers.

Development

git clone https://github.com/mihailorama/scrapefold.git
cd scrapefold
pip install -e ".[dev]"

# Pre-commit gate (lint + type-check + offline tests)
./scripts/check.sh

# Run tests
pytest -m "not paid and not network"

# Run live smoke (network, no API keys needed)
python scripts/live_smoke.py --max-pages 5

See CONTRIBUTING.md for engine-addition workflow and docs/workflows/development.md for the full dev loop.

Comparison

Engines — price & features

Sorted cheapest-first. The cost column is scrapefold's internal per-1000-call estimate (EngineCapabilities.estimated_cost_usd) — the figure the router's budget walks against. These are coarse placeholders for routing decisions, not official quotes; verify against each vendor's current pricing page before relying on them.

SaaS · site = ships site-specialized endpoints (LinkedIn, Google Maps, …). SaaS · search = query-driven search index rather than a page fetcher (the target is a search query or a blog URL). jina, keenable, and cloakbrowser set requires_api_key=False; keys are optional and raise free-tier rate limits.

SERP APIs

Scrapefold's public search() delegates to Enrichfold's 11 search engines. The table below compares specialist SERP APIs that are not part of that registry; several URL vendors also expose separate SERP endpoints.

Prices are approximate per-1000-search published rates and move between plan tiers — treat them as ballpark, not quotes.

Feature axes that matter when picking a SERP API: native result parsing (titles / links / snippets / ads / PAA as JSON vs. raw HTML), localization (geo_location + locale + device), supported engines beyond Google, and async batch vs. realtime latency.

Documentation

• docs/README.md — full documentation index
• docs/architecture/overview.md — module map, data flow, escalation ladder
• docs/conventions/golden-rules.md — invariants every engine adheres to
• docs/migration-guide.md — migrate from a hand-rolled cascade in four passes
• docs/tools/agent-mode.md — CLI + MCP server reference
• docs/TECH_DEBT.md — known limitations and follow-up roadmap

License

MIT. See LICENSE.

Note: Engine adapters are optional extras. SaaS engines require their own API keys (set via SCRAPEFOLD_<ENGINE>_API_KEY env vars); local engines have their own licenses — Scrapling (BSD), Crawl4AI (Apache), selenium (Apache), cloakbrowser (MIT). Scrapefold itself is MIT.