hamro-scrapy-playwright

Use scrapy-playwright with a custom stealth Chromium build.

Features

• Automatic scrapy-playwright patching
• Automatic browser download and management
• Drop-in replacement for existing scrapy-playwright projects
• Optional Cloudflare Turnstile auto-click integration

Install

pip install hamro-scrapy-playwright

Example

import scrapy
from scrapy import Request


class ExampleSpider(scrapy.Spider):
    name = "example"

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
            "https": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "PLAYWRIGHT_BROWSER_TYPE": "chromium",
        "PLAYWRIGHT_LAUNCH_OPTIONS": {
            "headless": False,
        },
        "PLAYWRIGHT_PROCESS_REQUEST_HEADERS": None,
    }

    async def start(self):
        yield Request(
            "https://www.example.com/",
            meta={
                "playwright": True,
                "playwright_include_page": True,
            },
            callback=self.parse,
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]

        try:
            await page.wait_for_load_state("networkidle")

            await page.evaluate(
                """
                () => {
                    document.body.style.backgroundColor = "#225577";
                }
                """
            )

            await page.wait_for_timeout(4000)

            self.logger.info("Title: %s", await page.title())

        finally:
            await page.close()

Cloudflare Turnstile Auto-Click

To enable automatic Turnstile clicking, enable HAMRO_CF_AUTOCLICK and configure a persistent Playwright context.

Headful mode (recommended for development)

import tempfile

custom_settings = {
    "DOWNLOAD_HANDLERS": {
        "http": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
        "https": "hamro_scrapy_playwright.handler.HamroPlaywrightDownloadHandler",
    },
    "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    "PLAYWRIGHT_BROWSER_TYPE": "chromium",
    "PLAYWRIGHT_LAUNCH_OPTIONS": {
        "headless": False,
    },
    "PLAYWRIGHT_CONTEXTS": {
        "persistent": {
            "user_data_dir": tempfile.mkdtemp(),
        }
    },
    "HAMRO_CF_AUTOCLICK": True,
    "HTTPERROR_ALLOWED_CODES": [403, 429],
}

A visible browser window opens and the bundled cf-autoclick extension runs, so you can watch the Turnstile widget get clicked in real time.

Headless mode

If running headless, use Chromium's new headless mode (extensions do not load in the old headless mode):

"PLAYWRIGHT_LAUNCH_OPTIONS": {
    "headless": True,
    "args": ["--headless=new"],
}

Verify the extension is active

The extension registers a background service worker in the browser context. Check it from your spider:

for worker in page.context.service_workers:
    if "chrome-extension" in worker.url:
        self.logger.info("cf-autoclick extension is ACTIVE: %s", worker.url)

You should also see cf-autoclick extension enabled: <path> in the Scrapy logs at startup.

Why a persistent context?

The Cloudflare auto-click integration loads as a Chromium extension. Chromium extensions require a persistent browser profile (user_data_dir), so HAMRO_CF_AUTOCLICK only works when using a persistent Playwright context. PLAYWRIGHT_LAUNCH_OPTIONS are automatically merged into the persistent context by this package.