This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Use a random User-Agent provided by fake-useragent for every request

Project Description

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent. It picks up User-Agent strings based on usage statistics from a real world database.

Installation

The simplest way is to install it via pip:

pip install scrapy-fake-useragent

Configuration

Turn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.

In Scrapy >=1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

In Scrapy <1.0:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

Configuring User-Agent type

There’s a configuration parameter RANDOM_UA_TYPE defaulting to random which is passed verbatim to the fake-user-agent. Therefore you can set it to say firefox to mimic only firefox browsers. Most useful though would be to use desktop or mobile values to send desktop or mobile strings respectively.

Usage with scrapy-proxies

To use with middlewares of random proxy such as scrapy-proxies, you need:

  1. set RANDOM_UA_PER_PROXY to True to allow switch per proxy
  2. set priority of RandomUserAgentMiddleware to be greater than scrapy-proxies, so that proxy is set before handle UA

Configuring Fake-UserAgent fallback

There’s a configuration parameter FAKEUSERAGENT_FALLBACK defaulting to None. You can set it to a string value, for example Mozilla or Your favorite browser, this configuration can completely disable any annoying exception.

Release History

Release History

This version
History Node

1.1.0

History Node

1.0.4

History Node

1.0.2

History Node

0.0.2

History Node

0.0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
scrapy_fake_useragent-1.1.0-py2.py3-none-any.whl (5.3 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Jun 16, 2017
scrapy-fake-useragent-1.1.0.tar.gz (2.9 kB) Copy SHA256 Checksum SHA256 Source Jun 16, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting