scrapy-proxy-rotator 0.1.0

Scrapy downloader middleware that rotates proxies.

This is a Scrapy downloader middleware that sets proxy server for requests. Currently proxy servers are rotated randomly. In the future more rotation strategies will be supported.

Configuration

Scrapy settings file:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_proxy_rotator.ProxyMiddleware': 1,
}

PROXY_ROTATOR = {
    'username': 'user1',
    'password': 'pass1',
    'proxies_file': 'proxies.txt',
}

In proxies file proxy server must be formatted like this http://host:port.