scrapy-crawio

Scrapy downloader middleware for Crawio.

Your spiders make normal requests; this middleware routes them through the Crawio API, which fetches each one from a healthy exit — using the cheapest route that works, up to a full browser when the site's anti-bot needs one — and returns the site's own response. No spider changes, no per-request proxy juggling. Only successful requests are billed.

Install

pip install scrapy-crawio

Configure (settings.py)

DOWNLOADER_MIDDLEWARES = {
    "scrapy_crawio.CrawioMiddleware": 1000,
}

CRAWIO_API_KEY = "rk-..."                   # required — dashboard -> API Key
CRAWIO_DOMAINS = ["example.com"]            # hosts to route; omit to route every request
CRAWIO_API_URL = "https://api.crawio.com"   # optional
CRAWIO_TIMEOUT = 120                        # seconds, optional
CRAWIO_SESSION = "job-1"                    # optional: default sticky session id
CRAWIO_COUNTRY = "DE"                       # optional: fetch every request from this country

# Recommended: let Scrapy retry the transient answers.
RETRY_ENABLED = True
RETRY_HTTP_CODES = [429, 502, 503, 504]

Use

Write spiders exactly as normal — the matched hosts go through Crawio:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://example.com/catalogue/page-1.html"]

    def parse(self, response):
        self.logger.info("fetched in %s ms", response.headers.get("X-Crawio-Ms", b"").decode())
        for href in response.css("h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item)

A page or JSON arrives as a TextResponse. A file (PDF, image, archive) arrives as a plain scrapy.http.Response with its original bytes in response.body.

Response headers

Every response through Crawio carries these, alongside the site's own headers:

Error responses (429, 502, 503, 504) carry X-Crawio-Request-Id and X-Crawio-Cost too, and a 429 carries Retry-After.

Per-request overrides

scrapy.Request(url, meta={"crawio_skip": True})       # download this one directly, not through us
scrapy.Request(url, meta={"crawio_session": "job-2"}) # pin to a sticky session
scrapy.Request(url, meta={"crawio_country": "FR"})    # fetch this one from France

A sticky session keeps the same exit and the same cookies across requests: sign in once, reuse the id, and the following requests stay signed in.

Countries

Fetch a page the way a visitor in another country sees it: local prices, local search results, pages that only open inside a country. Set a two-letter ISO 3166-1 code (case-insensitive) for the whole crawl, for one request, or both:

# settings.py: every request from Germany
CRAWIO_COUNTRY = "DE"

# one request from Brazil, one with no country (the best exit for the site)
yield scrapy.Request("https://example.com/precos", meta={"crawio_country": "BR"})
yield scrapy.Request("https://example.com/about", meta={"crawio_country": ""})

Leave the country out unless the page depends on it: Crawio then picks the best exit for each site. A country Crawio cannot serve answers 400 with error: country_unavailable before anything runs, and it is not billed. With a sticky session, keep the same country on every request of the session. Every country, and how the API answers, is in the docs.

Retries and billing

Every request carries an Idempotency-Key. When the connection to Crawio drops or times out before an answer arrives, the middleware raises scrapy_crawio.CrawioUnreachable (an OSError, which Scrapy's RetryMiddleware retries), and the retry carries the same key: it reattaches to the request already running instead of starting, and billing, a second one. An answer Crawio has returned (a 502 or 504, say) is final for its key, so a retry of it is sent as a new request with a new key. A 503 keeps its key. Errors are never billed, so retrying them never costs a credit.

What Crawio answers

Everything except 401 comes back as a real Response, so RetryMiddleware decides what to do with it and the body tells you why. Until 1.2.0 a retried 502/504 was refused with 409 (it reused the finished request's key) and a dropped connection was not retried at all.

Scrapy versions

Scrapy 2.6 and later. Since 1.2.1 the middleware's process_request is a coroutine and takes no spider argument, so Scrapy 2.13 and later print no deprecation warning for it (1.2.0 printed one when the crawl started and one per request on Scrapy 2.19).

License

MIT