Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

A middleware to change user-agent in request for Scrapy

Project description

Overview

Scrapy is a great framework for web crawling. This downloader middleware provides a user-agent rotation based on the settings in settings.py, spider, request.

Requirements

  • Tests on Python 2.7 and Python 3.5, but it should work on other version higher then Python 3.3
  • Tests on Linux, but it’s a pure python module, it should work on other platforms with official python supported, e.g. Windows, Mac OSX, BSD

Installation

The quick way:

pip install scrapy-useragents

Or put this middleware just beside the scrapy project.

Documentation

In setting.py, for example:

# -----------------------------------------------------------------------------
# USER AGENT
# -----------------------------------------------------------------------------

DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_useragents.downloadermiddlewares.useragents.UserAgentsMiddleware': 500,
})

USER_AGENTS = [
    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/57.0.2987.110 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/61.0.3163.79 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) '
     'Gecko/20100101 '
     'Firefox/55.0')  # firefox
]

Settings Reference

USER_AGENTS

A list of User-Agent to use when crawling, unless overridden.

The middleware will rotate this list by function cycle from the module itertools.

Be careful this middleware can’t handle the situation that the COOKIES_ENABLED is True, and the website binds the cookies with User-Agent, it may cause unpredictable result of the spider. This problem will be solved in the future.

TODO

  • Read User-Agent from a backend, e.g. MongoDB, MySQL, or even a file saved on the local disk.
  • Rotate User-Agent binding with cookies, keep the consistence
  • Add meta key for User-Agent selection based on each request

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Scrapy-UserAgents, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size Scrapy_UserAgents-0.0.1-py2.py3-none-any.whl (5.8 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size Scrapy-UserAgents-0.0.1.tar.gz (4.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page