scrapy-crawio

Scrapy downloader middleware for Crawio.

Your spiders make normal requests; this middleware routes them through the Crawio API, which fetches each one from a healthy exit — using the cheapest route that works, up to a full browser when the site's anti-bot needs one — and returns the site's own response. No spider changes, no per-request proxy juggling. Only successful requests are billed.

Install

pip install scrapy-crawio

Configure (settings.py)

DOWNLOADER_MIDDLEWARES = {
    "scrapy_crawio.CrawioMiddleware": 1000,
}

CRAWIO_API_KEY = "rk-..."                   # required — dashboard -> API Key
CRAWIO_DOMAINS = ["example.com"]            # hosts to route; omit to route every request
CRAWIO_API_URL = "https://api.crawio.com"   # optional
CRAWIO_TIMEOUT = 120                        # seconds, optional
CRAWIO_SESSION = "job-1"                    # optional: default sticky session id

# Recommended: let Scrapy retry the transient answers.
RETRY_ENABLED = True
RETRY_HTTP_CODES = [429, 502, 503, 504]

Use

Write spiders exactly as normal — the matched hosts go through Crawio:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"

    def start_requests(self):
        yield scrapy.Request("https://example.com/catalogue/page-1.html", callback=self.parse)

    def parse(self, response):
        self.logger.info("fetched in %s ms", response.headers.get("X-Crawio-Ms", b"").decode())
        for href in response.css("h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item)

A page or JSON arrives as a TextResponse. A file (PDF, image, archive) arrives as a plain scrapy.http.Response with its original bytes in response.body.

Response headers

Every response through Crawio carries these, alongside the site's own headers:

Error responses (429, 502, 503, 504) carry X-Crawio-Request-Id and X-Crawio-Cost too, and a 429 carries Retry-After.

Per-request overrides

scrapy.Request(url, meta={"crawio_skip": True})       # download this one directly, not through us
scrapy.Request(url, meta={"crawio_session": "job-2"}) # pin to a sticky session

A sticky session keeps the same exit and the same cookies across requests: sign in once, reuse the id, and the following requests stay signed in.

Retries and billing

Every request carries an Idempotency-Key, so a retry reattaches to the same job instead of starting — and billing — a second one. The key is stamped on the Request, so Scrapy's own RetryMiddleware reuses it.

What Crawio answers

Everything except 401 comes back as a real Response, so RetryMiddleware decides what to do with it and the body tells you why.

License

MIT