scraper

Web scraping service with anti-bot detection and Jina fallback. Runs as a standalone service or installs as a Python package.

Install

pip install dungle-scrubs-scraper

API

POST /scrape {"url": "https://example.com"}
GET  /health

Response

{
  "markdown": "# Page Title\n\nClean content...\n",
  "source": "crawl4ai",
  "url": "https://example.com",
  "bot_detected": false,
  "error": null,
  "elapsed_ms": 1234.5,
  "attempts": ["crawl4ai attempt 1"]
}

Pipeline

1. Crawl4AI — local Playwright headless Chromium, retries on transient errors
2. Bot detection — Cloudflare/CAPTCHA phrases + partial JS render heuristic (empty table cells)
3. Jina Reader fallback — free cloud re-fetch when bot-blocked or crawl fails
4. Artifact stripping — regex patterns for "Skip to content", GitHub chrome, duplicate lines

CLI

The scraper command exposes two subcommands:

scraper scrape <url>       # scrape a URL, JSON on stdout
scraper scrape --stdin     # read {"url": "..."} from stdin
scraper serve              # start the HTTP server

JSON output (stdout)

Every scrape invocation writes exactly one JSON object to stdout:

{
  "ok": true,
  "markdown": "# Page Title\n\nClean content...\n",
  "source": "crawl4ai",
  "url": "https://example.com",
  "bot_detected": false,
  "error": null,
  "elapsed_ms": 1234.5,
  "attempts": ["crawl4ai attempt 1"]
}

On failure, ok is false and error contains the reason. The structure is always the same — callers can unconditionally json.loads(stdout).

Exit codes

Code	Meaning
0	Success — content scraped
1	Scrape failed — all backends exhausted
2	Invalid input — bad URL or malformed stdin JSON
3	Timeout — global deadline exceeded
4	Internal error — unexpected crash

Timeout

The --timeout flag (default: 120s) wraps the entire pipeline in a global deadline. When exceeded, the process exits with code 3 and the JSON error field explains what happened.

scraper scrape --timeout 30 https://slow-site.example.com

stdin mode

Pipe a JSON object with a url field:

echo '{"url": "https://example.com"}' | scraper scrape --stdin

All logging goes to stderr. stdout is exclusively for the JSON result.

Run locally

uv sync
uv run scraper serve

Run with Docker

docker compose up -d

Environment

Variable	Required	Description
SCRAPER_PORT	No	Server port (default: 8020)
SCRAPER_HOST	No	Bind address (default: 127.0.0.1, local only; set 0.0.0.0 to expose)
SCRAPER_API_KEY	No	When set, POST /scrape requires a matching Authorization: Bearer or X-API-Key header
SCRAPER_JINA_ENABLED	No	Enable the third-party Jina Reader fallback (default: false)
SCRAPER_SERVER_TIMEOUT_SECONDS	No	Global per-request deadline for /scrape (default: 120)
SCRAPER_MAX_CONCURRENT_SCRAPES	No	Max concurrent scrapes before 503 (default: 4)
SCRAPER_CRAWL_TIMEOUT_SECONDS	No	Crawl4AI timeout (default: 25)

Security

This service fetches arbitrary, caller-supplied URLs and treats every inbound URL and every fetched page as untrusted. Defaults are tuned for a local, single-user tool:

• SSRF guard. Every URL — on the crawl4ai path, the Jina path, the CLI, and every redirect hop — is validated by a single guard that allows only http(s) and rejects hosts resolving to loopback, private, link-local, reserved, multicast, CGNAT, or cloud-metadata addresses (e.g. 127.0.0.1, 169.254.169.254, 10.0.0.0/8). Credentials in user:pass@host URLs are stripped before any request.
• Local by default. The server binds 127.0.0.1. Set SCRAPER_HOST=0.0.0.0 to expose it — and set SCRAPER_API_KEY when you do.
• Optional auth. With SCRAPER_API_KEY set, POST /scrape requires a matching Authorization: Bearer <key> or X-API-Key: <key> header.
• Resource bounds. A global per-request deadline, a concurrency cap (503 when exceeded), and response/markdown size caps protect against hangs and memory exhaustion.
• Third-party egress is opt-in. The Jina Reader fallback is disabled by default (SCRAPER_JINA_ENABLED=false) because it sends the target URL to a third party (r.jina.ai).

Report vulnerabilities privately — see SECURITY.md.

Use as a library

from scraper.scraper import scrape

result = await scrape("https://example.com")
print(result.markdown)

Use as a service

All consumers need one env var:

SCRAPER_URL=http://localhost:8020

import httpx

resp = httpx.post(f"{SCRAPER_URL}/scrape", json={"url": url})
data = resp.json()
markdown = data["markdown"]

Tests

uv run pytest tests/ -v

All mocked - no network required.

Port

8020 (registered in the shared port table).

License

MIT