Skip to main content

A middleware to change user-agent in request for Scrapy

Project description


Scrapy is a great framework for web crawling. This downloader middleware provides a user-agent rotation based on the settings in, spider, request.


  • Tests on Python 2.7 and Python 3.5, but it should work on other version higher then Python 3.3

  • Tests on Linux, but it’s a pure python module, it should work on other platforms with official python supported, e.g. Windows, Mac OSX, BSD


The quick way:

pip install scrapy-useragents

Or put this middleware just beside the scrapy project.


In, for example:

# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------

    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_useragents.downloadermiddlewares.useragents.UserAgentsMiddleware': 500,

    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/57.0.2987.110 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Linux x86_64) '
     'AppleWebKit/537.36 (KHTML, like Gecko) '
     'Chrome/61.0.3163.79 '
     'Safari/537.36'),  # chrome
    ('Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) '
     'Gecko/20100101 '
     'Firefox/55.0')  # firefox

Settings Reference


A list of User-Agent to use when crawling, unless overridden.

The middleware will rotate this list by function cycle from the module itertools.

Be careful this middleware can’t handle the situation that the COOKIES_ENABLED is True, and the website binds the cookies with User-Agent, it may cause unpredictable result of the spider. This problem will be solved in the future.


  • Read User-Agent from a backend, e.g. MongoDB, MySQL, or even a file saved on the local disk.

  • Rotate User-Agent binding with cookies, keep the consistence

  • Add meta key for User-Agent selection based on each request

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Scrapy-UserAgents-0.0.1.tar.gz (4.5 kB view hashes)

Uploaded Source

Built Distribution

Scrapy_UserAgents-0.0.1-py2.py3-none-any.whl (5.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page