ScrapeUnblocker Scrapy middleware

A Scrapy downloader middleware that routes your spider's requests through the ScrapeUnblocker web scraping API. Keep your spiders as they are: every page is fetched in a real browser behind ScrapeUnblocker's anti-bot bypass (Cloudflare, DataDome, PerimeterX, Akamai and others), and your callbacks receive the HTML as if the spider had downloaded the page itself.

• response.url, response.follow(), relative links and selectors work against the target site, not the API
• Per-request options through Request.meta, defaults in settings.py
• The target site's own 404 / 410 reaches your spider as a normal 404 / 410
• API errors keep their status and are explained in the log; 429 and 5xx go through Scrapy's RetryMiddleware

Install

pip install scrapeunblocker-scrapy-middleware

Requires Python 3.9+ and Scrapy 2.8+.

Get your API key at app.scrapeunblocker.com. The free trial does not require a credit card.

Enable it

In your project's settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapeunblocker_middleware.ScrapeUnblockerMiddleware": 543,
}

SCRAPEUNBLOCKER_API_KEY = "YOUR_API_KEY"  # or set the SCRAPEUNBLOCKER_API_KEY env var

# Optional: API options applied to every request
SCRAPEUNBLOCKER_DEFAULTS = {
    "proxy_country": "us",
}

# Recommended (see "Recommended settings" below)
CONCURRENT_REQUESTS = 20  # at most your plan's concurrent requests
CONCURRENT_REQUESTS_PER_DOMAIN = 20
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 523, 524, 408, 429]

That is all. Run your spider as usual.

What the middleware does

For every request it routes, the middleware:

1. Swaps the request for POST https://api.scrapeunblocker.com/getPageSource?url=<original URL> with your options in the query string. Booleans are sent as true / false, steps as a JSON array.
2. Sends your key in the x-scrapeunblocker-key header. The target site's own headers and cookies are not sent to the API.
3. Keeps the request's callback, errback, cb_kwargs, meta and priority.
4. When the answer arrives, gives the response the original URL and an original-looking response.request, so selectors, response.urljoin(), response.follow() and the referer of follow-up requests all point at the target site.
5. Reports the target's own status from the X-Origin-Status header as response.status (see Status codes).

Requests that are already addressed to the API host are never wrapped again, so retries are safe. The API call is rescheduled with dont_filter=True, so the dupefilter and the offsite filter (which already passed the original request) do not drop it, and it keeps the target site as its download slot, so CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY and DOWNLOAD_SLOTS still apply per target site.

Settings

Settings can also be set per spider with custom_settings.

API options

These are the /getPageSource options, set in SCRAPEUNBLOCKER_DEFAULTS or per request:

Browser rendering always happens; there is no option for it. Any other key is sent as-is, so an option added to the API later works without upgrading this package, but a name the API does not know has no effect, so the middleware logs a warning for it once (for example country -> use proxy_country). url cannot be set as an option: the request's own URL is sent.

Per-request options

Put a dict in Request.meta["scrapeunblocker"]. It is merged over SCRAPEUNBLOCKER_DEFAULTS for that request only; a value of None removes a default.

import scrapy


class PriceSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/product/123"]

    def parse(self, response):
        yield {"price": response.css(".price::text").get(default="").strip()}

        yield scrapy.Request(
            "https://example.com/product/124",
            meta={"scrapeunblocker": {"proxy_country": "de"}},
            callback=self.parse,
        )

Options are not inherited: a request made with response.follow(href) gets the defaults again unless you pass meta yourself.

Sending a request directly

To fetch a request without ScrapeUnblocker, set either of:

yield scrapy.Request(url, meta={"scrapeunblocker": False})
yield scrapy.Request(url, meta={"dont_scrapeunblocker": True})

Some requests are always sent directly, because /getPageSource cannot serve them:

• Requests that are not GET (for example FormRequest posts). The API loads a URL in a browser; it cannot replay a form submission. They go out directly and the middleware logs a warning once per method. To submit a form through ScrapeUnblocker, load the page with steps that fill in and submit the form.
• Requests Scrapy components make for themselves (callback=NO_CALLBACK): robots.txt when ROBOTSTXT_OBEY is on, and FilesPipeline / ImagesPipeline downloads. /getPageSource returns HTML documents only (a file or image answers 415), and routing robots.txt would cost a call per site. Setting meta["scrapeunblocker"] to a dict or True routes such a request anyway.
• URLs that are not http / https (data:, file:, ...).

Parsed data

With parsed_data the body is JSON, and response.json() works as usual:

def parse(self, response):
    payload = response.json()
    product = payload["data"]["data"]
    yield {"title": product.get("title"), "price": product.get("price")}

The shape, exactly as the API returns it:

{
  "data": {
    "page_type": "product_detail",
    "data": {
      "title": "A Light in the Attic",
      "price_incl_tax": "...",
      "images": ["https://..."]
    }
  }
}

• data.page_type is the detected page type (product_detail, listing_detail, article_detail, search_list, homepage, ..., not_found or unknown).
• data.data holds the extracted fields; list pages put results in items with total and next_page.
• When the page rendered but nothing could be extracted, the answer is still 200, data.data is empty, data_extracted is false and html carries the rendered page so you can parse it yourself.
• A dead URL comes back as 404 / 410 with {"data": {"page_type": "not_found", "data": {}}}.

To know in a callback which options a response was fetched with (defaults included), read response.meta["scrapeunblocker_params"]:

def parse(self, response):
    if response.meta["scrapeunblocker_params"].get("parsed_data"):
        yield response.json()["data"]["data"]
    else:
        yield {"title": response.css("h1::text").get(default="").strip()}

Field reference: parsed data guide.

Browser steps

steps runs actions in a real browser after the page loads, then returns the resulting HTML. Use it to wait for content that appears late, click "load more", scroll, or search:

yield scrapy.Request(
    "https://example.com/search",
    meta={
        "scrapeunblocker": {
            "steps": [
                {"action": "type", "selector": "input[name=q]", "value": "headphones"},
                {"action": "press_key", "value": "Enter"},
                {"action": "wait_for", "selector": ".results"},
            ]
        }
    },
)

Actions: wait_for, wait_for_text, wait, click, type, select, press_key, scroll. A step that cannot run answers 422 with a JSON body naming the step; the middleware logs it as error=step_failed, reason=..., step_index=..., action=.... A request with steps runs its actions every time it is sent, retries included. Full reference: browser steps.

Cookies

With get_cookies the body is JSON: html (the page), cookies (every cookie set during the visit) and proxy_address. Read it with response.json():

def parse(self, response):
    payload = response.json()
    selector = scrapy.Selector(text=payload["html"])
    cookies = {c["name"]: c["value"] for c in payload["cookies"]}

What your callback receives

• response.url is the URL you requested. The API follows the target's redirects itself and does not report the final URL, so on a page reached through a redirect, relative links resolve against the URL you requested.
• response.request is a GET for that URL with your callback, headers, cookies and meta.
• response.headers are the API's headers, including X-Origin-Status when the target answered with an error of its own.
• response.flags contains "scrapeunblocker", so the Crawled (200) <GET ...> log line shows which pages came through the API.
• The HTML is re-indented by the API, so text nodes carry surrounding whitespace and newlines. Strip them: response.css("h1::text").get(default="").strip(), normalize-space() in XPath, or MapCompose(str.strip) in an item loader.

response.meta gains these keys:

Status codes and errors

The target site's own answer. When the API reached the site and the site itself answered with an error, the response carries X-Origin-Status. The middleware makes that the response status, so Scrapy treats it the way it would treat the site's answer:

These calls are billed and retrying returns the same answer, so the middleware marks them dont_retry. HttpErrorMiddleware drops them unless you allow the status, for example meta={"handle_httpstatus_list": [404, 410]}. The first one of each status is logged at INFO. Set SCRAPEUNBLOCKER_ORIGIN_STATUS = False to keep the API's status instead.

API errors keep their status and are logged with the API's message and a hint (first occurrence of each at WARNING, or ERROR for 401 / 402; repeats at DEBUG):

Failed calls are not billed, so retrying them costs nothing. The exceptions are 400 Hostname cannot be resolved (the domain does not exist) and a 422 from a failed browser step (the page was rendered). Full list: errors.

Recommended settings

# Your plan's concurrent requests limit, not more: requests over it answer 429.
CONCURRENT_REQUESTS = 20
# The API call keeps the target site as its download slot, so this caps how
# many calls run at once for one site. Raise it to use your plan's limit on a
# single-site crawl (Scrapy's default is 8).
CONCURRENT_REQUESTS_PER_DOMAIN = 20

# A call can take its render time cap plus queueing time when the API is busy.
# Keep Scrapy's default; lower values cancel calls that would have succeeded.
DOWNLOAD_TIMEOUT = 180

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 523, 524, 408, 429]

# Calls take seconds, so AutoThrottle (which adapts to latency) slows a crawl
# down a lot. Cap throughput with the concurrency settings above instead.
AUTOTHROTTLE_ENABLED = False

Concurrency limits per plan: concurrency limits.

About the priority: give the middleware a number higher than RobotsTxtMiddleware's 100, so robots.txt rules are checked against the target site before the request is rewritten (the middleware warns at startup otherwise). 543, just below RetryMiddleware (550), also lets RetryMiddleware see the API's own status before the target's status is applied.

Stats

The middleware adds these counters to Scrapy's stats:

Calling other endpoints

A request you address to the API host yourself is passed through untouched, and the middleware adds the API key header if it is missing. That lets a spider call other endpoints, for example Google results from /serpApi:

yield scrapy.Request(
    "https://api.scrapeunblocker.com/serpApi?keyword=web+scraping",
    method="POST",
    callback=self.parse_serp,
)

Links

• ScrapeUnblocker
• Dashboard and API key
• Documentation
• Python SDK for calling the API outside Scrapy

License

MIT - see LICENSE.