Scrapy download handler that integrates aiohttp
Project description
scrapy-aiohttp-downloader
scrapy-aiohttp-downloader
is a Scrapy download handler.
Installation
pip install scrapy-aiohttp-downloader
Activation
Replace the default http
and/or https
Download Handlers through DOWNLOAD_HANDLERS
DOWNLOAD_HANDLERS = {
"http": "scrapy_aiohttp_downloader.AioHTTPDownloadHandler",
"https": "scrapy_aiohttp_downloader.AioHTTPDownloadHandler",
}
Also, be sure to install the asyncio-based Twisted reactor:
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
Basic usage
Set the aiohttp
Request.meta key to download a request using aiohttp
:
import scrapy
class AioHTTPSpider(scrapy.Spider):
name = "spider"
custom_settings = {
"DOWNLOAD_HANDLERS": {
"http": "scrapy_aiohttp_downloader.AioHTTPDownloadHandler",
"https": "scrapy_aiohttp_downloader.AioHTTPDownloadHandler",
},
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
def start_requests(self):
yield scrapy.Request(
"https://example.com/",
meta={"aiohttp": True},
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-aiohttp-downloader-1.0.0b1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f50fa3ad65d4d82ce21d1a94a1ee6fec2b8c30ab4ce0564b1d8ed54967db703f |
|
MD5 | 804ec611f9c82187e9e992857e10014a |
|
BLAKE2b-256 | cb16a5f8a97122af4ddf87719a7fb4f5ac1483b2a12a8d43d7312ca84e479b12 |
Close
Hashes for scrapy_aiohttp_downloader-1.0.0b1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc9ccc6b2c8c18a9da331ae3cd584971db55f4bd0a28f9b0b1ca7981d99c73ce |
|
MD5 | a4862bda5840469b5499410b28560070 |
|
BLAKE2b-256 | 8e38ca7c57c210251d11bf4cb4904a3c58b2bcb07746a48ad71af465277541e3 |