proxy-middleware 0.2.0

Scrapy http proxy middleware that gets proxy parameters from settings

Scrapy middlewares that reads proxy config from settings.

Contents

• Install

• ProxyFromSettingsMiddleware

• ProxyOnlyTorMiddleware

• License

  • History

• 0.2.0 (2017-05-29)

• 0.1.1 (2017-04-07)

Install

pip install proxy-middleware

ProxyFromSettingsMiddleware

A middleware that sets proxy from settings file.

Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:

DOWNLOADER_MIDDLEWARES = {
   'proxy_middleware.ProxyFromSettingsMiddleware': 10,
   ...

Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:

scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118

ProxyOnlyTorMiddleware

A middleware that proxies only requests to TOR sites (domain ends with “.onion”).

Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:

DOWNLOADER_MIDDLEWARES = {
   'proxy_middleware.ProxyOnlyTorMiddleware': 10,
   ...

Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.

License

License is MIT.

History

0.2.0 (2017-05-29)

• ProxyOnlyTorMiddleware added.

0.1.1 (2017-04-07)

• Initial release