scrape-plan

Before you write a scraper, find out which of three approaches this page actually needs — and whether you're allowed to.

$ scrape-plan https://quotes.toscrape.com/scroll

https://quotes.toscrape.com/scroll

  robots.txt   allowed   no robots.txt
  body in HTML no        0% of the visible content is already in the raw HTML

  Tier 1  Call the internal JSON API
  This page calls a JSON endpoint directly. Hitting it without a browser is the
  fastest option and survives redesigns far better than parsing markup.

  internal JSON API (1)
    https://quotes.toscrape.com/api/quotes?page=1
      GET 200 · application/json  seen 19x
      shape {has_next, page, quotes, tag, top_ten_tags}
      pagination query parameter ?page=

# ready-to-run code follows...

That page has 95 characters of HTML and 16,816 characters on screen. Every rendering-based tool concludes "client-side rendered, you need a browser" and you accept ~2 seconds per page. But the page is just calling a JSON API, and calling it directly takes ~0.1 seconds. Over 1,000 pages that is 33 minutes versus 1 minute 40.

Nothing told you the API was there. That is the gap this fills.

The three tiers

Checking 1 and 2 before reaching for 3 is the highest-leverage decision in extraction work. Most people skip straight to 3 because nothing tells them not to.

Install

pip install scrape-plan[browser]
playwright install chromium

The browser is only needed to detect internal APIs (tier 1) — that requires watching what the page actually requests. Without it, scrape-plan --no-browser still evaluates robots.txt and can confirm tier 2 when the body is already in the raw HTML. When the raw HTML is nearly empty it reports Tier ? rather than guessing, because at that point the page is either JavaScript-rendered or a block page and there is no way to tell from the HTML alone:

pip install scrape-plan     # no browser, no Playwright download

Usage

scrape-plan https://example.com/products      # full analysis + code
scrape-plan URL --code                        # only the generated code
scrape-plan URL --json                        # machine-readable
scrape-plan URL --no-browser                  # skip tier-1 detection
scrape-plan URL --scrolls 6                   # more scrolling for lazy loaders

Exit codes, so it composes in scripts:

scrape-plan "$URL" --no-browser >/dev/null || echo "skipping $URL"

It tells you when not to scrape

robots.txt is evaluated for your exact path, with correct group boundaries, wildcards, $ anchors, and longest-match-wins precedence. The rule that matched is printed, so you can check the verdict rather than trust it.

$ scrape-plan "https://apps.shopify.com/search?q=test" --no-browser

  robots.txt   DISALLOWED   Disallow: *q=*
  ...
  This path is disallowed. Do not crawl it.

  No code generated. Re-run with --force if you have permission.

No runnable code is generated for a disallowed path. A tool that flags the problem and then hands you the scraper anyway has not actually flagged anything.

Generated code

The code is not a sketch. It runs, and it carries the things people add last and therefore never add: a request interval and a User-Agent with a contact address.

The tier-2 snippet reads resp.content, never resp.text. When a server declares no charset, requests falls back to ISO-8859-1 per RFC 2616 and silently corrupts non-Latin text — no exception is raised, the data is just wrong, and you find out much later from someone who can read the language.

What it is not

This does not scrape anything for you. Tools like Scrapy, Crawlee and Firecrawl do that well. This runs before them and answers one question: given this page, what is the cheapest approach that works, and am I permitted to use it?

It also does not defeat bot protection, solve CAPTCHAs, or access anything behind a login.

How it works

1. Fetches robots.txt from the target's origin and evaluates your exact path.
2. Fetches the raw HTML and measures how much of the rendered page is already present in it, by comparing three-word sequences rather than raw lengths. Comparing lengths does not work: hidden menus and templates inflate the raw side, which made one server-rendered page look 4x larger than its own rendered output.
3. Opens the page in Chromium, scrolls a few times to trigger lazy requests, and records the XHR/fetch responses that actually carry content. Analytics, error reporting, ad trackers, cookie-consent config and tag managers are filtered out — one page alone served Sentry, an ad tracker, OneTrust and Zaraz, and reporting any of those as "the internal API" would send you reverse-engineering the wrong endpoint.
4. Groups endpoints by parameter names rather than values, so a paginated API hit 19 times shows up once rather than 19 times.
5. Picks the tier and emits matching code. If the body is already in the raw HTML, that wins over any API it found: you do not need to reverse-engineer an endpoint to read something the server already sent you.

At most 8 endpoints are probed, so the tool never becomes a source of load on the site being inspected.

Tests

python3 tests/test_core.py     # 24 cases, no network

The robots parser and the endpoint grouping carry most of the tests, because both fail silently: a wrong answer looks exactly like a right one.

Related

A Chrome extension with the same core, for pages you already have open: scrape-plan extension

License

MIT © Jinhyuk Sung