scrapy-fake-useragent 1.0.2

Use a random User-Agent provided by fake-useragent every request

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent. It picks up User-Agent strings based on usage statistics from a real world database.

Configuration

Turn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.

In Scrapy >=1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

In Scrapy <1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

Usage with scrapy-proxies

To use with middlewares of random proxy such as scrapy-proxies, you need:

1. set RANDOM_UA_PER_PROXY to True to allow switch per proxy

2. set priority of RandomUserAgentMiddleware to be greater than scrapy-proxies, so that proxy is set before handle UA