scrapy-scylla-proxies 0.5.0.5

scrapy-scylla-proxies: Random proxy middleware for Scrapy that fetches valid proxies from Scylla.

Random proxy middleware for Scrapy

Using Scylla to fetch valid proxies.

---

NOTE: I am not a 'real' programmer, help always appreciated! But it works! ... for now.

Processes Scrapy requests using a random proxies to avoid IP ban and improve crawling speed, this plugs in to the Scylla project which provides a local database of proxies.

Install & run Scylla

The Scylla project will need to be set-up separately!! The quickest way to do this is to use the docker container. The following command will download and run Scylla (provided you have docker installed of course).

docker run -d -p 8899:8899 -p 8081:8081 --name scylla wildcat/scylla:latest

Install scrapy-scylla-proxies

The quick way:

pip install scrapy-scylla-proxies

Or checkout the source and run

python setup.py install

What to put in Scrapy's 'settings.py'

This is stuff you are going to need to integrate this middleware with Scrapy.

SSP_ENABLED - This MUST be set to True.

SSP_SCYLLA_URI - The location of the Scylla API (Default: 'http://localhost:8899').

SSP_PROXY_TIMEOUT - How often the proxy list is refreshed (Default: 60s).

SSP_HTTPS - Whether to only use HTTPS proxies, You will need this set to True if you are scraping an HTTPS site (Default: True).

SSP_SPLASH_REQUEST_ENABLED - Whether this middleware will need to set the proxy for a 'scrapy.Request' or a 'SplashRequest' (Default: False)

Example 'settings.py'

This is a sample taken directly from a working scraper of mine, I used it to scrape approximately 15000 items from a website without any 'bans'.

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    # For retries
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 290,
    # For random scylla proxies
    'scrapy_scylla_proxies.random_proxy.RandomProxyMiddleware': 300,
    # For http proxy ip rotation
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

DOWNLOAD_TIMEOUT = 180
RETRY_TIMES = 10
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 400, 429, 403, 404]

# scrapy-scylla-proxies settings
# Enabled
SSP_ENABLED = True
# Location of the scylla server
SSP_SCYLLA_URI = 'http://localhost:8899'
# Proxy timeout in seconds
SSP_PROXY_TIMEOUT = 60
# Get only https proxies
SSP_HTTPS = True

Tips

I also find that rotating your user agent in combination with this middleware can be helpful in minimising failures due to being banned!

Donate

If you like this middleware or it was helpful to you, you can always send me a small donation, even just a token amount. It will encourage me to keep developing this middleware and improving it! :::fire:::

Donate here!