Skip to main content

advanced-scrapy-proxies

advanced-scrapy-proxies is a Python library for dealing with proxies in your Scrapy project. Starting from Aivarsk's scrapy proxy (no more updated since 2018) i'm adding more features to manage lists of proxies generated dinamically.

Installation

Use the package manager pip to install advanced-scrapy-proxies.

pip install advanced-scrapy-proxies

Usage

settings.py

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'advanced-scrapy-proxies.RandomProxy': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110
}

## Proxy mode
	# -1: NO_PROXY, middleware is configured but does nothing. Useful when needed to automate the selection of the mode
	# 0: RANDOMIZE_PROXY_EVERY_REQUESTS, every requrest use a different proxy
	# 1: RANDOMIZE_PROXY_ONCE, selects one proxy for the whole execution from the input list
	# 2: SET_CUSTOM_PROXY, use the proxy specified with option CUSTOM_PROXY
	# 3: REMOTE_PROXY_LIST, use the proxy list at the specified URL
PROXY_MODE = 3

PROXY_LIST ='https://yourproxylisturl/list.txt'

As every scrapy project, you can override the settings in settings.py when calling the scraper.

##PROXY_MODE=-1, the spider does not use the proxy list provided.
scrapy crawl myspider -s PROXY_MODE=-1 -s PROXY_LIST='myproxylist.txt'
##PROXY_MODE=0, the spider use the proxy list provided, choosing for every request a different proxy. 
scrapy crawl myspider -s PROXY_MODE=0 -s PROXY_LIST='myproxylist.txt'
##PROXY_MODE=1, the spider use the proxy list provided, choosing only one proxy for the whole execution.
scrapy crawl myspider -s PROXY_MODE=1 -s PROXY_LIST='myproxylist.txt'
##PROXY_MODE=2, the spider uses the proxy provided.
scrapy crawl myspider -s PROXY_MODE=2 -s PROXY_LIST='http://myproxy.com:80'
##PROXY_MODE=3, the spider uses the proxy list at the url provided. The list is read at every request made by the spider, so it can be updated during the execution.
scrapy crawl myspider -s PROXY_MODE=3 -s PROXY_LIST='http://myproxy.com:80'

Planned new features and updates

Minor updates

  • adding more tests on the format of the input variables
  • rewriting error messages

New features

  • Adding a cooldown list: instead of deleting proxy after a failed attempt to get data, use a cooldown list where they are not used for a limited time in the scraper but ready to be reused when the cooldown finishes.
  • Adding support for reading urls of the lists behind user and password
  • Updating proxy list at every request even for PROXY_MODE=0

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

GNU GPLv2

Metadata

Release files for advanced-scrapy-proxies 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for advanced-scrapy-proxies 0.1.3
File Size Uploaded Provenance Actions
advanced-scrapy-proxies-0.1.3.tar.gz 10.8 kB Without provenance Details for advanced-scrapy-proxies-0.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for advanced-scrapy-proxies 0.1.3
File Interpreter ABI Platform Actions
advanced_scrapy_proxies-0.1.3-py3-none-any.whl Python 3 none any Details for advanced_scrapy_proxies-0.1.3-py3-none-any.whl

Total release size: 22.7 kB

Release files / advanced-scrapy-proxies-0.1.3.tar.gz

Download URL advanced-scrapy-proxies-0.1.3.tar.gz
Size 10.8 kB
Tags Source
SHA-256 checksum
How to use checksums
52356685740fb5de3a57f5c93a8353eead6e860be883453229559debdf0650ac
BLAKE2b-256 checksum
How to use checksums
df456f04b34de1e8eadbe8a95a229753e13c2de85199dd8afb9e2850b0223881
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.0 CPython/3.7.9

Release files / advanced_scrapy_proxies-0.1.3-py3-none-any.whl

Download URL advanced_scrapy_proxies-0.1.3-py3-none-any.whl
Size 11.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6101306a9a2577574bb524d3da7cb7b9f007532745cc6236a40dcf82c6d3ac0d
BLAKE2b-256 checksum
How to use checksums
0d7946e415c6ecfb64250ac0003af627696fb2d434546b032cfbc22055534242
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.0 CPython/3.7.9

Security for advanced-scrapy-proxies 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page