Skip to main content

A middleware to change user-agent in request for Scrapy

Project description

Overview

Scrapy is a great framework for web crawling. This downloader middleware provides a user-agent rotation based on the settings in settings.py, spider, request.

Requirements

  • Tests on Python 2.7 and Python 3.5, but it should work on other version higher then Python 3.3
  • Tests on Linux, but it’s a pure python module, it should work on other platforms with official python supported, e.g. Windows, Mac OSX, BSD

Installation

The quick way:

pip install scrapy-useragents

Or put this middleware just beside the scrapy project.

Documentation

In setting.py, for example:

# -----------------------------------------------------------------------------
# USER AGENT
# -----------------------------------------------------------------------------

DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_useragents.downloadermiddlewares.useragents.UserAgentsMiddleware': 500,
})

USER_AGENTS = [
    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/57.0.2987.110 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/61.0.3163.79 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) '
     'Gecko/20100101 '
     'Firefox/55.0')  # firefox
]

Settings Reference

USER_AGENTS

A list of User-Agent to use when crawling, unless overridden.

The middleware will rotate this list by function cycle from the module itertools.

Be careful this middleware can’t handle the situation that the COOKIES_ENABLED is True, and the website binds the cookies with User-Agent, it may cause unpredictable result of the spider. This problem will be solved in the future.

TODO

  • Read User-Agent from a backend, e.g. MongoDB, MySQL, or even a file saved on the local disk.
  • Rotate User-Agent binding with cookies, keep the consistence
  • Add meta key for User-Agent selection based on each request

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Scrapy-UserAgents, version 0.0.1
Filename, size & hash File type Python version Upload date
Scrapy_UserAgents-0.0.1-py2.py3-none-any.whl (5.8 kB) View hashes Wheel py2.py3
Scrapy-UserAgents-0.0.1.tar.gz (4.5 kB) View hashes Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page