Use a random User-Agent provided by fake-useragent for every request
Project description
Random User-Agent middleware based on fake-useragent. It picks up User-Agent strings based on usage statistics from a real world database.
Installation
The simplest way is to install it via pip:
pip install scrapy_fake_useragent
Configuration
Turn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.
In Scrapy >=1.0:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
In Scrapy <1.0:
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
Configuring User-Agent type
There’s a configuration parameter RANDOM_UA_TYPE defaulting to random which is passed verbatim to the fake-user-agent to random choose user agents. random, chrome, firefox, safari, internetexplorer are supported. If you want to choose from a specific device type, you can use a device prefix before browse type, such as desktop.chrome, mobile.chrome, only desktop, mobile, tablet are supported.
Usage with scrapy-proxies
To use with middlewares of random proxy such as scrapy-proxies, you need:
set RANDOM_UA_PER_PROXY to True to allow switch per proxy
set priority of RandomUserAgentMiddleware to be greater than scrapy-proxies, so that proxy is set before handle UA
Configuring Fake-UserAgent fallback
There’s a configuration parameter FAKEUSERAGENT_FALLBACK defaulting to None. You can set it to a string value, for example Mozilla or Your favorite browser, this configuration can completely disable any annoying exception.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scrapy_fake_useragent_fix-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b6626324646016610505717628739a12cfc9828eced66beff7a058f1fccf595 |
|
MD5 | d82e854436f6a83bf3e6f981d87d050b |
|
BLAKE2b-256 | b4eadfeda14979491715ce1a37fcca81770ddfe1c7016ed1653c3815dec50d89 |