🔥 PyreCrawl — Self-Hosted Firecrawl Alternative as an MCP Server

A self-hosted Firecrawl alternative combining two best-in-class open-source scrapers behind a single Model Context Protocol (MCP) server:

• Scrapling — fast HTTP (curl_cffi) + stealth browser with Cloudflare Turnstile bypass, adaptive element tracking, and XHR capture
• Crawl4AI — LLM-first crawler: BM25 fit-markdown, citations, structured extraction, deep crawl (BFS/DFS/BestFirst)

A smart auto-fallback ladder tries the cheapest engine that succeeds:

fast HTTP (Scrapling)
    │  (403/503/Cloudflare challenge or empty body)
    ▼
stealth browser (Scrapling StealthyFetcher + CF solver)
    │  (still blocked or page needs full JS rendering)
    ▼
full LLM processing (Crawl4AI AsyncWebCrawler + BM25)

⚡ Tools exposed

prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (full Crawl4AI).

🚀 Install & Use (one-liner)

1. Install

# Using uv (recommended — fast, isolated, no venv needed)
uv tool install pyrecrawl

# Or pipx (alternative)
pipx install pyrecrawl

# Or pip into a venv
pip install pyrecrawl

2. One-time browser engines

pyrecrawl setup

This installs Playwright Chromium + Scrapling engines (~2 min, one-time).

3. Register with your AI agent

# Auto-detect installed agents and write their MCP configs
pyrecrawl install

# Or target specific agents
pyrecrawl install claude-desktop cursor

# Dry-run to preview what would change
pyrecrawl install --dry-run

Supported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.

4. Start chatting

After installing + registering, restart your agent (or start a new session). Then ask:

"Scrape https://example.com and summarize it."

The tools appear as mcp_pyrecrawl_scrape, mcp_pyrecrawl_extract, mcp_pyrecrawl_map_site, mcp_pyrecrawl_crawl, mcp_pyrecrawl_search, mcp_pyrecrawl_health.

📚 Manual config (if pyrecrawl install doesn't match your setup)

Windows note: uvx must be on PATH. If not, use the full path to uvx.exe (e.g. C:\Users\<you>\AppData\Local\hermes\bin\uvx.exe).

🧠 Why two engines?

PyreCrawl = Scrapling for fetch & bypass + Crawl4AI for processing & extraction.

📊 Compared to Firecrawl (hosted)

🔧 Development

git clone https://github.com/SanggonBoy/PyreCrawl.git
cd PyreCrawl
uv venv --python 3.12 .venv
source .venv/Scripts/activate  # Windows; or .venv/bin/activate on macOS/Linux
uv pip install -e ".[dev]"
python -m playwright install chromium
scrapling install

Run tests

python scripts/selfcheck.py   # real-network smoke test
python scripts/probe_stdio.py # stdio JSON-RPC probe

📦 Publish

Maintainers only:

git tag v0.2.1
git push origin v0.2.1

GitHub Actions builds + uploads to PyPI via trusted publishing.

📜 Uninstall

# Remove from all agent configs
pyrecrawl uninstall

# Remove the package
uv tool uninstall pyrecrawl

🛡️ License

MIT — see LICENSE.