scrapy-tor-downloader 1.0.0

Scrapy middleware with TOR support for more robust scrapers or anonymous scraping.

scrapy-tor-downloader

Scrapy middleware with TOR support for more robust scrapers or anonymous scraping.

Dependencies :globe_with_meridians:

• Python 3.7
• Scrapy 2.6.1
• torpy 1.1.6

Installation :inbox_tray:

This is a python package hosted on pypi, so to install simply run the following command:

pip install scrapy-tor-downloader

Settings

TOR_PROXY_ENABLED

Whether TOR is used to proxy any request (defaults to false).

Meta field to enable/disable this per request is: tor_proxy_enabled

TOR_FALLBACK_ENABLED

Whether TOR is used when a request fails as a fallback (defaults to true).

Meta field to enable/disable this per request is: tor_fallback_enabled

Usage example :eyes:

In order to use this plugin simply add the following settings and substitute your variables:

DOWNLOADER_MIDDLEWARES = {
    "tormiddleware.middleware.TORDownloaderMiddleware": 631
}

This will immediately allow you begin using TOR as a fallback when one of your requests fail. In order to use it as a proxy you can add the following to your settings:

TOR_PROXY_ENABLED = True

This will make every request hit TOR for a response. If you have turned the proxy on the TOR fallback is ignored, however if it is off the fallback is still on by default, which means if a request returns an error it will be tried again on TOR. In order to turn this off add the following to your settings:

TOR_FALLBACK_ENABLED = False

License :memo:

The project is available under the MIT License.