scrapy-rnet 0.0.3

Scrapy download handler that can impersonate browser TLS signatures

scrapy-rnet

A Scrapy download handler that enables browser TLS fingerprint impersonation using the rnet library.

Features

• Seamlessly integrates with Scrapy's download system
• Impersonates browser TLS signatures to evade detection
• Supports multiple browser versions for impersonation

Installation

pip install scrapy-rnet

Requirements

• Python 3.12+
• rnet >= 2.1.0
• scrapy >= 2.12.0

Usage

Basic Setup

1. Configure the download handlers in your Scrapy settings:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_rnet.handler.ImpersonateDownloadHandler",
    "https": "scrapy_rnet.handler.ImpersonateDownloadHandler",
}
USER_AGENT = None

# Required for async operation
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

2. Enable impersonation for specific requests by adding impersonate to the request metadata:

from rnet import Impersonate, ImpersonateOS

yield scrapy.Request(
    url="https://example.com",
    meta={
        "impersonate": Impersonate.Chrome132,
        "impersonate_os": ImpersonateOS.Windows # Optional
    },
    callback=self.parse
)

Example Spider

import scrapy
from scrapy.http import Request, Response
from rnet import Impersonate, ImpersonateOS

class BrowserImpersonationSpider(scrapy.Spider):
    name = "browser_impersonation"
    
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_rnet.handler.ImpersonateDownloadHandler",
            "https": "scrapy_rnet.handler.ImpersonateDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "USER_AGENT": None
    }
    
    def start_requests(self):
        url = "https://httpbin.org/headers"
        yield Request(
            url=url,
            meta={
                "impersonate": Impersonate.Chrome132,
                "impersonate_os": ImpersonateOS.Windows # Optional
            },
            callback=self.parse
        )
    
    def parse(self, response):
        self.logger.info(response.text)

Available Impersonation Options

The package supports a wide range of browser versions and operating systems:

Browser Versions

Browser	Versions
Chrome	Chrome100, Chrome101, Chrome104, Chrome105, Chrome106, Chrome107, Chrome108, Chrome109, Chrome114, Chrome116, Chrome117, Chrome118, Chrome119, Chrome120, Chrome123, Chrome124, Chrome126, Chrome127, Chrome128, Chrome129, Chrome130, Chrome131, Chrome132, Chrome133, Chrome134
Edge	Edge101, Edge122, Edge127, Edge131, Edge134
Safari	SafariIos17_2, SafariIos17_4_1, SafariIos16_5, Safari15_3, Safari15_5, Safari15_6_1, Safari16, Safari16_5, Safari17_0, Safari17_2_1, Safari17_4_1, Safari17_5, Safari18, SafariIPad18, Safari18_2, Safari18_1_1, Safari18_3, Safari18_3_1
OkHttp	OkHttp3_9, OkHttp3_11, OkHttp3_13, OkHttp3_14, OkHttp4_9, OkHttp4_10, OkHttp4_12, OkHttp5
Firefox	Firefox109, Firefox117, Firefox128, Firefox133, Firefox135, FirefoxPrivate135, FirefoxAndroid135, Firefox136, FirefoxPrivate136

Operating Systems

• Windows
• MacOS
• Linux
• Android
• IOS

Credits

0x676e67 for Rnet