PXM2 Scrapy Middleware & Playwright Rotation Helper

Official Downloader Middleware and browser automation helper for integrating PXM2 4G/5G Rotating Mobile Proxies with Scrapy and Playwright.

For an in-depth tutorial and production scraping patterns, read our comprehensive Scrapy Web Scraping Guide.

Features

• Automatic Proxy Injection: Route all Scrapy requests through PXM2 high-reputation mobile proxy gateways with authentication.
• Smart Ban Recovery: Detect blocks (403 Forbidden, 429 Too Many Requests, or connection timeouts) and trigger instant modem IP rotation before retrying the request.
• Frequency-Based Rotation: Rotate cellular IPs automatically every N requests.
• Hardware Cooldown Handling: Built-in delay allowing cellular modems to complete radio disconnection and obtain a fresh carrier CGNAT IP.
• Playwright Rotation Helper: Ready-to-use rotator for headless Chromium/Firefox and scrapy-playwright.
• Zero Heavy Dependencies: Pure standard library implementation.

Installation

pip install pxm2-scrapy

Or install from source:

git clone https://github.com/pxm2/pxm2-scrapy-middleware.git
cd pxm2-scrapy-middleware
pip install -e .

Scrapy Setup

1. Enable Middleware in settings.py

Add PXM2ProxyMiddleware to your Scrapy project's DOWNLOADER_MIDDLEWARES and configure your credentials:

# settings.py

DOWNLOADER_MIDDLEWARES = {
    'pxm2_scrapy.PXM2ProxyMiddleware': 610,
}

# PXM2 Mobile Proxy Credentials
PXM2_PROXY_HOST = 'proxy.pxm2.io'
PXM2_PROXY_PORT = 10001
PXM2_PROXY_USERNAME = 'your_username'
PXM2_PROXY_PASSWORD = 'your_password'

# Instant IP rotation webhook URL from your PXM2 dashboard
PXM2_ROTATION_URL = 'https://api.pxm2.io/rotate/YOUR_API_TOKEN'

# HTTP status codes that trigger automatic IP rotation and request retry
PXM2_ROTATE_ON_STATUS = [403, 429, 503]

# Rotate every N requests (0 to disable scheduled rotation)
PXM2_ROTATE_EVERY = 50

# Hardware reconnection cooldown in seconds (default: 5.0)
PXM2_ROTATION_COOLDOWN = 5.0

# Max retries per request after rotation
PXM2_MAX_RETRIES = 3

Scrapy Spider Example

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = [
        "https://httpbin.org/ip",
        "https://example.com/products",
    ]

    def parse(self, response):
        self.logger.info("Fetched %s via IP: %s", response.url, response.text.strip())
        # Parse product details...

Playwright Integration

Use PXM2PlaywrightRotator with standalone Playwright or scrapy-playwright:

from playwright.sync_api import sync_playwright
from pxm2_scrapy import PXM2PlaywrightRotator

# Initialize rotator
rotator = PXM2PlaywrightRotator(
    host="proxy.pxm2.io",
    port=10001,
    username="your_username",
    password="your_password",
    rotation_url="https://api.pxm2.io/rotate/YOUR_API_TOKEN",
    cooldown=5.0
)

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy=rotator.get_proxy_config(),
        headless=True
    )
    context = browser.new_context()
    page = context.new_page()

    page.goto("https://api.ipify.org?format=json")
    print("Initial IP:", page.content())

    # Trigger instant carrier IP rotation
    print("Rotating mobile IP...")
    rotator.rotate()

    page.reload()
    print("New IP after rotation:", page.content())

    browser.close()

Configuration Reference

Why PXM2 4G/5G Mobile Proxies?

Traditional datacenter and residential proxies are increasingly identified and blocked by anti-bot systems (Cloudflare Turnstile, DataDome, PerimeterX, Akamai).

PXM2 Mobile Proxies leverage real SIM modems connected to tier-1 mobile carriers. Because millions of real smartphone users share the same mobile carrier CGNAT IP pools, mobile proxies achieve the highest trust scores and bypass anti-scraping protections effortlessly.

Learn more:

• PXM2 Rotating Mobile Proxies
• Scrapy Integration Guide
• Documentation & Tutorials
• Contact Support

License

MIT License. See LICENSE for details.