███████╗██╗    ██╗██╗████████╗ ██████╗██╗  ██╗██████╗  █████╗  ██████╗██╗  ██╗
██╔════╝██║    ██║██║╚══██╔══╝██╔════╝██║  ██║██╔══██╗██╔══██╗██╔════╝██║ ██╔╝
███████╗██║ █╗ ██║██║   ██║   ██║     ███████║██████╔╝███████║██║     █████╔╝
╚════██║██║███╗██║██║   ██║   ██║     ██╔══██║██╔══██╗██╔══██║██║     ██╔═██╗
███████║╚███╔███╔╝██║   ██║   ╚██████╗██║  ██║██████╔╝██║  ██║╚██████╗██║  ██╗
╚══════╝ ╚══╝╚══╝ ╚═╝   ╚═╝    ╚═════╝╚═╝  ╚═╝╚═════╝ ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝

Why

Most scrapers either give up on hard pages or send everything through an expensive headless browser / paid API. switchback orders the methods by cost and walks them cheapest-first, per host, learning which tier wins where so the next run starts there. The easy majority stays free; only genuinely-walled hosts pay for the heavy tiers.

• Cost-ordered cascade — free APIs → cheap HTTP → anti-bot solver → stealth browser → paid API.
• Per-host memory (botwall) — remembers the winning tier per host, skip-lists hard blockers, auto-skips hosts stuck on the paid tier.
• Cost-scoped residential egress — routes only walled hosts through a residential proxy, never the easy majority.
• One shape, three entry points — Python library, CLI (JSON on stdout), or an HTTP service.
• Observable — every attempt is an OpenTelemetry span; logs ship trace-correlated to any OTLP backend (Jaeger, Tempo, SigNoz).
• Runs with any subset installed — each tier imports its deps lazily; a missing one is just a tier miss.

Quickstart

pip install switchback                 # core: cheap tiers (0/1) + search

from switchback import scrape

for r in scrape(["https://arxiv.org/abs/1706.03762"]):
    print(r.source_method, len(r.markdown))

python -m switchback https://example.com/article    # JSON on stdout — bridge for any language

That's the whole loop. Add tiers as you need them (see Install).

_{A real run, time-compressed. Run 1 climbs the cascade until a tier gets through. Run 2 hits the same host, and botwall starts at the tier that won last time. The paid tier is never touched.}

The cascade (stop at first success)

Every URL has a wall-clock budget (SCRAPER_DEADLINE_S, default 45s) checked between tiers so one URL can't run the whole cascade of timeouts. Each tier attempt records latency + outcome (ok / short_content / rate_limited / miss / not_applicable) to its span and the botwall event log; the root span carries total latency and the final outcome (incl. deadline_exceeded).

Search (query → URLs) is separate from the scrape cascade: switchback.search() / python -m switchback.api --search <query>, backed by a local SearXNG.

Install

pip install switchback                 # core: normalization + cheap tiers (tier_1/tier_2) + search
pip install "switchback[cloudflare]"   # + tier_3 Cloudflare/anti-bot solver (cloudscraper)
pip install "switchback[server]"       # + HTTP service (fastapi, uvicorn) incl. /metrics + /traces
pip install "switchback[browser]" && patchright install chromium   # + tier_4 stealth Chromium
pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
pip install "switchback[all]"          # all of the above
pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)

mcp, youtube and documents are not part of [all], so upgrading an existing [all] install doesn't pull new dependencies (documents brings markitdown and onnxruntime). Add them by name. YouTube transcripts and office files also need their switch (SCRAPER_YOUTUBE=1, SCRAPER_DOCUMENTS=1), because they change what those URLs return; without it they behave exactly as before, installed or not.

For tier_3's full v3 JS-VM + Turnstile + stealth, install the Enhanced Edition 3.x fork (PyPI's cloudscraper is the older v1/v2 — PyPI forbids pinning a git-URL dep inside a published package, so install it alongside):

pip install "cloudscraper @ git+https://github.com/VeNoMouS/cloudscraper@3.0.0"

Or run the whole thing as a container: docker build -t switchback . && docker run -p 8799:8799 switchback.

Production / cold-start deployment

The two heavy tiers pull dependencies that often can't be baked into a base image and land after boot (e.g. an async install thread on Azure). Until they're ready, those tiers report unavailable (a distinct outcome carrying the exact fix) and the cascade falls through — they are never silently skipped. Checklist:

• tier_4 is the real workhorse for Cloudflare/JS sites — make sure its browser is installed: patchright install chromium (note: patchright, not vanilla playwright). On a cold start, run this in your post-boot install step/thread; tier_4 flips to ready once it finishes.
• tier_3 needs the cloudscraper 3.x fork (above) to attempt stealth. With the frozen PyPI cloudscraper it reports unavailable and fails fast (no wasted solve budget) instead of erroring mid-cascade. tier_3 is a weak solver for modern Cloudflare — treat it as a cheap try before the browser, not the primary.
• Install Node.js for tier_3's v3 JS-VM challenges — faster and thread-safe vs. the pure-Python js2py fallback (relevant under concurrent load).
• Bound tier_3's solve budget with SCRAPER_TIER_3_TIMEOUT_S (default 25; the old SCRAPER_CLOUDSCRAPER_TIMEOUT_S is still honored) so an unsolvable challenge can't eat the per-URL deadline before the browser tier runs. Lower it (e.g. 12) if tier_3 rarely wins on your hosts.

Scraping URLs other people submit

If the URLs come from users you don't fully trust (a "save this link" box, a feed reader, a public API), two opt-in switches harden the engine. Both default to off, so nothing changes unless you set them:

export SCRAPER_DISABLE_CLOUDSCRAPER=1   # tier_3 runs site JavaScript outside a browser sandbox
export SCRAPER_BROWSER_BLOCK_PRIVATE=1  # browser tiers can't reach your machine's or network's private services

• Why tier_3: solving a Cloudflare JS challenge means running JavaScript the site sent, in js2py or Node's vm module, and neither is a security boundary. The browser tiers run page JavaScript inside the browser's own sandbox instead.
• Install tier_4's browser first (patchright install chromium): with tier_3 off, it is what clears Cloudflare. --doctor shows tier_3 as off and still exits 0.
• Why the browser guard: a scraped page can redirect, frame or fetch() an address like http://127.0.0.1:… or a cloud metadata endpoint, and whatever answers comes back as scraped content. With the guard, those requests (and WebSockets) are refused and service workers are disabled so they can't route around it. Check the URL you pass in yourself too: the guard covers what the page asks for in the browser tiers, not the HTTP tiers' own requests.

Verify readiness on the box with the preflight check (doubles as a healthcheck — exit 0 when the capable tiers are ready):

switchback --doctor          # or: python -m switchback --doctor

Use it from your app

Three interchangeable entry points — all return the same shape ([{url, source_method, markdown}], successes only):

Python library

from switchback import scrape
for r in scrape(["https://arxiv.org/abs/1706.03762"]):
    print(r.source_method, len(r.markdown))

# Need failures + reasons too? scrape_detailed returns a ScrapeOutcome per URL
# (ok, final_outcome, error_class, status_code, and the per-tier attempts):
from switchback import scrape_detailed
for o in scrape_detailed(["https://www.pcmag.com/news"]):
    if not o.ok:
        print(o.url, o.final_outcome, o.error_class, o.status_code)

CLI (JSON on stdout — bridge for any language)

python -m switchback https://example.com/article        # or: switchback <url>

HTTP service (language-agnostic; one warm process keeps the browser pool hot)

switchback-server                                    # listens on :8799
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"]}'
curl 'localhost:8799/search?q=web+scraping'

Non-Python callers: see clients/node_bridge.md. Python callers that want HTTP-with-CLI-fallback can drop in clients/python_client.py.

MCP server (for agents: Claude Code, Cursor, any MCP client)

pip install "switchback[mcp]"
claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search

Agent-ready extras

Off by default: without them every result is exactly what it was before, the same ScrapeResult / ScrapeOutcome with the same fields, and the same CLI and HTTP JSON. Ask for any combination per call; results then come back as ScrapeResultWithExtras / ScrapeOutcomeWithExtras (subclasses, so existing isinstance checks still hold) with the extras you asked for filled in and the rest None. The CLI and HTTP JSON add only the keys you asked for.

from switchback import scrape
[r] = scrape("https://example.com/product", state=True, appendix=True,
             query="what does it cost?")
r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]

switchback --state --appendix --query "what does it cost?" https://example.com/product
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'

state and appendix need the fetched HTML, so they skip the content cache (SCRAPER_CONTENT_TTL_S) and fetch; highlights work on cached content too.

Grading highlights. The default grader is local BM25 keyword scoring: free, offline, no data leaves the box. For better relevance, opt into TypeSafe's Jev model, which answers "does this passage answer the question?" for every passage and returns a calibrated probability (passages under SCRAPER_HIGHLIGHTS_MIN_SCORE, default 0.5, are dropped, so an off-topic page honestly returns []):

export SCRAPER_HIGHLIGHTS_GRADER=jev
export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai

Jev is billed per input token (about $0.0004 for a 10k-token page). If the call fails, that request falls back to BM25 ("grader": "bm25"); the scrape never fails because of highlights.

Cost-scoped residential egress

The dominant reason hard hosts wall you is the datacenter IP, not the fingerprint. When a host repeatedly walls the local tiers (a 403/429 or a bot-wall page, SCRAPER_BOTWALL_EGRESS_AFTER times) it's flagged needs_egress and the cascade reruns through a residential proxy — but only for that host:

export SCRAPER_EGRESS_PROXY="http://user:pass@p.webshare.io:80"

The easy majority that already succeeds free at the datacenter IP stays direct, so you never spend (often metered) residential bandwidth on it. Escalation tries the cheap HTTP tiers through the proxy first (~0.2MB/page) before the heavier browser tiers. Webshare's free plan includes ~1GB/mo of residential bandwidth — enough for low-volume hard-host recovery at $0. Use SCRAPER_PROXY instead to force every request through a proxy.

Metrics & reporting

The engine derives all metrics from its own state files (no external store): the botwall event log (one row per tier attempt, incl. the detected challenge vendor) and the per-host DB (winning tier, per-vendor challenge_counts).

curl localhost:8799/metrics            # cost savings vs Firecrawl, coverage,
                                       # overall + per-tier latency, outcomes
curl localhost:8799/metrics/domains    # per-domain: error codes, challenges, latency
python -m switchback.flags             # periodic digest: domains stuck on Firecrawl,
                                       # escalated to egress, top challenged (cron-friendly)

Both endpoints accept ?minutes=N to window the event-derived sections. The savings figure compares engine spend (Firecrawl invocations only) against a Firecrawl-everything baseline, charging the hard-page credit multiplier (BENCH_FIRECRAWL_HARD_MULT) for URLs that needed a browser/residential tier or hit a challenge — i.e. exactly the ones Firecrawl bills more for.

Configuration

All configuration is via environment variables. The engine runs with missing pieces: each tier imports its deps lazily and a missing one just counts as a tier miss. Tracing no-ops if OTel isn't installed/configured.

export OTEL_SERVICE_NAME=switchback
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Logged-in sessions

Beyond a static SCRAPER_COOKIES_FILE, wire SCRAPER_LOGIN_HOOK to a callable func(host) -> {cookie: value}. When an authenticated host trips a login/bot wall, the engine calls the hook once, persists the returned cookies per host, and overlays them on every tier (and future runs), then re-runs that URL on a fresh budget. The hook owns the site-specific login mechanics; the engine stays generic.

Session traces

With SCRAPER_TRACE_SESSION=1, each browser-tier attempt writes a Playwright trace zip to state/traces/. Manage them over HTTP — GET /traces (list), GET /traces/{id} (download), DELETE /traces/{id} — and open one with playwright show-trace <zip>. Off by default (traces are MBs each).

Output formats

Markdown is the default and is unchanged. Pick a different shape globally with SCRAPER_OUTPUT_FORMAT, or per call:

from switchback import scrape
scrape(["https://example.com/article"])                    # markdown (default)
scrape(["https://example.com/article"], fmt="html")        # raw HTML
scrape(["https://example.com/article"], fmt="markdown_trimmed")

switchback --format html_selectors https://example.com/article
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"format":"html"}'

The chosen content rides in the result's markdown field; in the CLI/server JSON the key is markdown for markdown formats and html for html formats. The API/PDF tiers (arXiv synth, PDF→text) have no HTML, so html formats fall back to their text for those sources.

Per-domain extraction

Markdown of the whole page is the default. To scope a site to its content node or strip site-specific noise, declare prefs per host in config/extraction.json (see config/extraction.example.json); every tier's normalize step picks them up automatically.

Roadmap

Not built yet:

• Layout-aware markdown — keep pricing grids and card layouts built from <div>s (not <table>s) as tables, using the browser tiers' rendered layout.
• Scanned pages and images — OCR / vision-model text for PDFs with no text layer and for images.
• Audio and caption-less video — speech-to-text transcription.
• Hosted service — managed browsers at high concurrency, stored results and shareable result links. switchback stays a library you run yourself.
• Smaller gaps: Next.js App Router flight data (self.__next_f) in state, and YouTube chapters in transcripts.

Contributing

Issues and PRs welcome — see CONTRIBUTING.md. Start with the cascade runner in switchback/orchestrator.py.

Responsible use

This engine is for lawful data collection. You are responsible for respecting each target site's Terms of Service, robots.txt, and rate limits, and for having the right to access the content you fetch. The stealth / anti-bot tiers (cloudscraper, patchright, camoufox) exist to handle legitimate access friction (e.g. generic bot interstitials on public pages) — not to evade access controls, paywalls, or authentication you aren't authorized to bypass. The software is provided "as is", without warranty (see LICENSE).

License

MIT — see LICENSE. Third-party dependencies and their licenses are listed in NOTICE; all are permissive (MIT / BSD-3-Clause / Apache-2.0) and compatible with this project's MIT license.