Scrapy middleware with requests support as an alternative to Twisted calls.
Project description
scrapy-requests-downloader
Scrapy middleware with requests support as an alternative to Twisted calls.
Dependencies :globe_with_meridians:
Installation :inbox_tray:
This is a python package hosted on pypi, so to install simply run the following command:
pip install scrapy-requests-downloader
Settings
REQUESTS_ENABLED
Whether the requests middle is used to process any request (defaults to false).
Meta field to enable/disable this per request is: requests_enabled
Usage example :eyes:
In order to use this plugin simply add the following settings and substitute your variables:
DOWNLOADER_MIDDLEWARES = {
"requestsmiddleware.middleware.RequestsDownloaderMiddleware": 632
}
In order to use it as a downloader you can add the following to your settings:
REQUESTS_ENABLED = True
This will make every request use requests for a response.
License :memo:
The project is available under the MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for scrapy-requests-downloader-1.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 739ae9243a7252c16f899493b74a08e4aa5e184a30a36f56af1a2f7ffcd75a8b |
|
MD5 | 0db04972006cb74e89bac59208ba5970 |
|
BLAKE2b-256 | c70c9680ad1654b5986731926a5bf80d3938bbcc193dc5ba56822c99b778f900 |