scrapy-hrequests

A Scrapy download handler that uses hrequests instead of Scrapy's default HTTP downloader. This allows you to use hrequests seamlessly within your existing Scrapy spiders without changing your crawling logic.

Features

• Simple integration with Scrapy
• Supports both HTTP and HTTPS requests
• Drop-in replacement for Scrapy's default download handler
• Uses the hrequests library for making requests

Installation

pip install scrapy-hrequests

Usage

Configure your spider (or project) to use the HRequestsDownloadHandler.

Per Spider

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_hrequests.handler.HRequestsDownloadHandler",
            "https": "scrapy_hrequests.handler.HRequestsDownloadHandler",
        },
    }

    start_urls = [
        "https://httpbin.org/get",
    ]

    def parse(self, response):
        yield {
            "url": response.url,
            "status": response.status,
        }

Project Wide

Add the following to your settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_hrequests.handler.HRequestsDownloadHandler",
    "https": "scrapy_hrequests.handler.HRequestsDownloadHandler",
}

Requirements

• Python 3.8+
• Scrapy
• hrequests

License

MIT License