scrapy-fake-useragent-fix 0.1.0

Use a random User-Agent provided by fake-useragent for every request

Random User-Agent middleware based on fake-useragent. It picks up User-Agent strings based on usage statistics from a real world database.

Installation

The simplest way is to install it via pip:

pip install scrapy_fake_useragent

Configuration

Turn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.

In Scrapy >=1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

In Scrapy <1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

Configuring User-Agent type

There’s a configuration parameter RANDOM_UA_TYPE defaulting to random which is passed verbatim to the fake-user-agent to random choose user agents. random, chrome, firefox, safari, internetexplorer are supported. If you want to choose from a specific device type, you can use a device prefix before browse type, such as desktop.chrome, mobile.chrome, only desktop, mobile, tablet are supported.

Usage with scrapy-proxies

To use with middlewares of random proxy such as scrapy-proxies, you need:

1. set RANDOM_UA_PER_PROXY to True to allow switch per proxy

2. set priority of RandomUserAgentMiddleware to be greater than scrapy-proxies, so that proxy is set before handle UA

Configuring Fake-UserAgent fallback

There’s a configuration parameter FAKEUSERAGENT_FALLBACK defaulting to None. You can set it to a string value, for example Mozilla or Your favorite browser, this configuration can completely disable any annoying exception.