rotating-free-proxies
Avoid being banned by webpages when you crawl them. This is an extension to the amazing scrapy-rotating-proxies library. The main target of this library is to get proxies dynamically when the spider is running. This library automatically fetches freely available lists of proxies from free-proxy-list.net.
Installation
pip install rotating-free-proxies
Usage
After installing you need to do just add following two variables in settings.py of your Scrapy project
ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt' # Path that this library uses to store list of proxies
NUMBER_OF_PROXIES_TO_FETCH = 5 # Controls how many proxies to use
DOWNLOADER_MIDDLEWARES = {
'rotating_free_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_free_proxies.middlewares.BanDetectionMiddleware': 620,
}
For further details on using this library, refer to the original readme.
Thank you!
CHANGES
0.1.1 (2020-05-28)
Fixes bug when user does not create file to store proxies. Adds settings to control number of proxies to use
0.1.0 (2020-05-01)
Initial release
1 maintainer
Release files for rotating-free-proxies 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| rotating-free-proxies-0.1.2.tar.gz | 8.8 kB | Details |
Release files / rotating-free-proxies-0.1.2.tar.gz
| Download URL | rotating-free-proxies-0.1.2.tar.gz |
|---|---|
| Size | 8.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3ccea9852bf068ed04dc9845ed6ab045299655099aa5b9d02be78ad9bd9df7f7
|
|
BLAKE2b-256 checksum How to use checksums |
40a006735efa4ce15bb0ff445e25647124ad7be4ddea0cf0db9622dd979f5f61
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
Python-urllib/3.7
|
