Middleware to easily implement ScraperAPI in Scrapy projects
Project description
scrapy-scraperapi-middleware
This middleware lets you use Scraper API for every request you process with Scrapy.
You could instead set the proxy
field of your request's meta attribute to http://scraperapi.your=options:your_key@proxy-server.scraperapi.com:8001
and enable HttpProxyMiddleware
.
This middleware's only purpose is to achieve this in a more convenient manner.
Installation
$ pip install scrapy-scraperapi-middleware
Settings
You need to specify your key for Scraper API in your settings.py or settings object.
SCRAPER_API_KEY = 'your_key'
You also need to enable ScraperAPIMiddleware as well as Scrapy's HttpProxyMiddleware.
DOWNLOADER_MIDDLEWARES = {
'your_project.middleware_file.ScraperAPIMiddleware': 350,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}
Additional options
Scraper API supports additional options as documented here. You could use these options by adding them to your settings as dictionary.
SCRAPER_API_OPTIONS = {
'render': 'true',
'country_code': 'us'
}
Affiliate link
If this example is helpful to you and you do not yet have a subscription to Scraper API, consider using my affiliate link if you plan on getting one. Be aware that all other links to ScraperAPI in this Readme are also affiliate links.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for scrapy-scraperapi-middleware-0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36fb5cd6dde7f77eddbf7fb5fdf1cb064a8d985f4e9581f5e61a1f4da809eba2 |
|
MD5 | 38ac7d28ed25644296a83845ca145c9c |
|
BLAKE2b-256 | f8313b9e2539101e3359dacb9299bec0a676431b4a91b0d1e1011418d4e74bf8 |
Hashes for scrapy_scraperapi_middleware-0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9890ce86969f0d556322784cb0b3cb76debf93c8d155b3272b549f3c46aa419 |
|
MD5 | 64f6be45de05907e772aa515e7631be8 |
|
BLAKE2b-256 | 9b2d03065ddf4586a9ce7e450e9572af8db832221418c4e52b00f409874ee062 |