scrapy-scraperapi-middleware 1.0

Middleware to easily implement ScraperAPI in Scrapy projects

scrapy-scraperapi-middleware

This middleware lets you use Scraper API for every request you process with Scrapy.
You could instead set the proxy field of your request's meta attribute to http://scraperapi.your=options:[e-mail]:8001 and enable HttpProxyMiddleware.
This middleware's only purpose is to achieve this in a more convenient manner.

Installation

$ pip install scrapy-scraperapi-middleware

Settings

You need to specify your key for Scraper API in your settings.py or settings object.

SCRAPERAPI_KEY = 'your_key'

You also need to enable ScrapyScraperAPIMiddleware as well as Scrapy's HttpProxyMiddleware.

DOWNLOADER_MIDDLEWARES = {
    'scrapy_scraperapi_middleware.ScrapyScraperAPIMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

Additional options

Scraper API supports additional options as documented here. You could use these options by adding them to your settings as dictionary.

SCRAPERAPI_OPTIONS = {
    'render': 'true', 
    'country_code': 'us'
}

Affiliate link

If this example is helpful to you and you do not yet have a subscription to Scraper API, consider using my affiliate link if you plan on getting one. Be aware that all other links to ScraperAPI in this Readme are also affiliate links.