Rotating proxies for Scrapy
Project description
rotating-free-proxies
Avoid being banned by webpages when you crawl them. This is an extension to the amazing scrapy-rotating-proxies library. The main target of this library is to get proxies dynamically when the spider is running. This library automatically fetches freely available lists of proxies from free-proxy-list.net.
Installation
pip install rotating-free-proxies
Usage
After installing you need to do just add following two variables in settings.py of your Scrapy project
ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt' # Path that this library uses to store list of proxies NUMBER_OF_PROXIES_TO_FETCH = 5 # Controls how many proxies to use DOWNLOADER_MIDDLEWARES = { 'rotating_free_proxies.middlewares.RotatingProxyMiddleware': 610, 'rotating_free_proxies.middlewares.BanDetectionMiddleware': 620, }
For further details on using this library, refer to the original readme.
Thank you!
CHANGES
0.1.1 (2020-05-28)
Fixes bug when user does not create file to store proxies. Adds settings to control number of proxies to use
0.1.0 (2020-05-01)
Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for rotating-free-proxies-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2488db2721239dbcf1578cee84880659576c52c4dec57c873756c9fe1d905f5d |
|
MD5 | 171906034b0e4a03f3a1662ce8a26e71 |
|
BLAKE2b-256 | ee189afb67c745f7cb537b6f4c78bc183ee062407ffa527def6ac4c8de62ec67 |