os-scrapy-httpproxy-middleware 0.0.2

Auto Generated by os-scrapy-cookiecutter

os-scrapy-httpproxy-middleware

This project provide middleware to enhance Scrapy built-in HttpProxyMiddleware. You can set proxy without sheme as the following format:

[user:password@]proxy:port

Install

pip install os-scrapy-httpproxy-middleware

You can run example spider directly in the project root path

scrapy crawl example

Usage

Settings

• enable middleware, it is better disable Scrapy built-in HttpProxyMiddleware

  DOWNLOADER_MIDDLEWARES = {
      "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
      "os_scrapy_httpproxy_middleware.middlewares.HttpProxyMiddleware": 750,
  }
  

Unit Tests

sh scripts/test.sh

License

MIT licensed.