Skip to main content

Scrapy Proxy Headers

PyPI version Documentation

Send custom headers to proxies and receive proxy response headers in Scrapy.

The Problem

When making HTTPS requests through a proxy, Scrapy cannot send custom headers to the proxy itself. This is because HTTPS requests create an encrypted tunnel (via HTTP CONNECT) - any headers you add to request.headers are encrypted and only visible to the destination server, not the proxy.

┌──────────┐     CONNECT      ┌───────┐     Encrypted     ┌────────────┐
│  Scrapy  │ ───────────────► │ Proxy │ ════════════════► │ Target URL │
└──────────┘  (unencrypted)   └───────┘    (tunnel)       └────────────┘
                  │                              │
           Proxy headers             request.headers
           go HERE                   go here (encrypted)

This extension solves the problem by:

  1. Sending custom headers to the proxy during the CONNECT handshake
  2. Capturing response headers from the proxy's CONNECT response
  3. Making those headers available in your spider

Installation

pip install scrapy-proxy-headers

Quick Start

1. Configure the Download Handler

In your Scrapy settings.py:

DOWNLOAD_HANDLERS = {
    "https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler"
}

Or in your spider's custom_settings:

class MySpider(scrapy.Spider):
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler"
        }
    }

2. Send Proxy Headers

Use request.meta["proxy_headers"] to send headers to the proxy:

import scrapy

class MySpider(scrapy.Spider):
    name = "example"
    
    def start_requests(self):
        yield scrapy.Request(
            url="https://api.ipify.org?format=json",
            meta={
                "proxy": "http://your-proxy:port",
                "proxy_headers": {"X-ProxyMesh-Country": "US"}
            }
        )
    
    def parse(self, response):
        # Proxy response headers are available in response.headers
        proxy_ip = response.headers.get("X-ProxyMesh-IP")
        self.logger.info(f"Proxy IP: {proxy_ip}")

3. Receive Proxy Response Headers

Headers from the proxy's CONNECT response are merged into response.headers. Headers the origin already sent are left unchanged.

def parse(self, response):
    # Access headers sent by the proxy
    proxy_ip = response.headers.get(b"X-ProxyMesh-IP")
    if proxy_ip:
        print(f"Request made through IP: {proxy_ip.decode()}")

Complete Example

import scrapy

class ProxyHeadersSpider(scrapy.Spider):
    name = "proxy_headers_demo"
    
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler"
        }
    }
    
    def start_requests(self):
        yield scrapy.Request(
            url="https://api.ipify.org?format=json",
            meta={
                "proxy": "http://us.proxymesh.com:31280",
                "proxy_headers": {"X-ProxyMesh-Country": "US"}
            },
            callback=self.parse_ip
        )
    
    def parse_ip(self, response):
        data = response.json()
        proxy_ip = response.headers.get(b"X-ProxyMesh-IP")
        
        self.logger.info(f"Public IP: {data['ip']}")
        if proxy_ip:
            self.logger.info(f"Proxy IP: {proxy_ip.decode()}")
        
        yield {
            "public_ip": data["ip"],
            "proxy_ip": proxy_ip.decode() if proxy_ip else None
        }

How It Works

  1. HTTP11ProxyDownloadHandler - Custom download handler that manages proxy header caching
  2. ScrapyProxyHeadersAgent - Agent that reads proxy_headers from request meta
  3. TunnelingHeadersAgent - Sends custom headers in the CONNECT request
  4. TunnelingHeadersTCP4ClientEndpoint - Captures proxy response headers from CONNECT response

The handler also caches proxy response headers by proxy URL. This ensures headers remain available even when Scrapy reuses existing tunnel connections for subsequent requests.

Test Harness

A test harness is included to verify proxy header functionality:

# Basic test
PROXY_URL=http://your-proxy:port TEST_URL=https://api.ipify.org python test_proxy_headers.py

# With custom proxy header
PROXY_URL=http://your-proxy:port \
PROXY_HEADER=X-ProxyMesh-IP \
SEND_PROXY_HEADER=X-ProxyMesh-Country \
SEND_PROXY_VALUE=US \
python test_proxy_headers.py

# Verbose output
python test_proxy_headers.py -v

Environment Variables

Variable Description Default
PROXY_URL Proxy URL (also checks HTTPS_PROXY) Required
TEST_URL URL to request https://api.ipify.org?format=json
PROXY_HEADER Response header to check for X-ProxyMesh-IP
SEND_PROXY_HEADER Header name to send to proxy Optional
SEND_PROXY_VALUE Value for the send header Optional

Documentation

Full documentation is available at scrapy-proxy-headers.readthedocs.io.

Use Cases

  • Geographic targeting: Send X-ProxyMesh-Country to route through specific countries
  • Session consistency: Request the same IP across multiple requests
  • Debugging: Capture proxy response headers to see which IP was assigned
  • Load balancing: Use proxy headers to control request distribution

Requirements

  • Python 3.8+
  • Scrapy >=2.17.0,<2.20

License

BSD License - see LICENSE for details.

Metadata

Release files for scrapy-proxy-headers 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-proxy-headers 0.2.2
File Size Uploaded Provenance Actions
scrapy_proxy_headers-0.2.2.tar.gz 7.8 kB With provenance Details for scrapy_proxy_headers-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrapy-proxy-headers 0.2.2
File Interpreter ABI Platform Actions
scrapy_proxy_headers-0.2.2-py3-none-any.whl Python 3 none any Details for scrapy_proxy_headers-0.2.2-py3-none-any.whl

Total release size: 16.0 kB

Release files / scrapy_proxy_headers-0.2.2.tar.gz

Download URL scrapy_proxy_headers-0.2.2.tar.gz
Size 7.8 kB
Tags Source
SHA-256 checksum
How to use checksums
3adacfb1a7d89502d774b06a446fbe4c0b5ad93706f586e7d6ad3415354f5a55
BLAKE2b-256 checksum
How to use checksums
7227d24cfb31fe3b46985c5511aadc0d4c3a2f2cddec07b1b324a3d9571b000f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / scrapy_proxy_headers-0.2.2-py3-none-any.whl

Download URL scrapy_proxy_headers-0.2.2-py3-none-any.whl
Size 8.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c4800e1350945bdcf4cf45f4b1bfc0e97f5c7bf76b9026778131742cf3afb1c3
BLAKE2b-256 checksum
How to use checksums
b889005338240935b78a9b95ec12f2f5cbec651e2747dbd050a7f4d080af6a2b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for scrapy-proxy-headers 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scrapy_proxy_headers-0.2.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page