Crawlera middleware for Scrapy
Project description
scrapy-crawlera provides easy use of Crawlera with Scrapy.
Installation
You can install scrapy-crawlera using pip:
pip install scrapy-crawlera
You can then enable the middleware in your settings.py:
DOWNLOADER_MIDDLEWARES = { ... 'scrapy_crawlera.CrawleraMiddleware': 600 }
Credentials
There are two ways to specify credentials.
Through settings.py:
CRAWLERA_ENABLED = True CRAWLERA_APIKEY = 'apikey'
Through spider attributes:
class MySpider: crawlera_enabled = True crawlera_apikey = 'apikey'
How to use it
You just need to specify the headers when making a request like:
scrapy.Request( 'http://example.com', headers={ 'X-Crawlera-Max-Retries': 1, ... }, )
Remember that you could also set which headers to use by default by all requests with DEFAULT_REQUEST_HEADERS
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy-crawlera-1.1.0.tar.gz
(3.3 kB
view hashes)
Built Distribution
Close
Hashes for scrapy_crawlera-1.1.0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23ca51b3fb4d8e7f912ed52591fb3134cb65e088778dd38e02226dcf0bd125ed |
|
MD5 | 47d792e7ac7d459a1b84ca0c37a612af |
|
BLAKE2b-256 | 5a229e8e1e98c440496d8cb9e4bfee0e7fe3d00cf2a6d26a43915b7815482b6a |