A mini spider framework, Integrate aiohttp into scrapy
Project description
Python async library for web scraping PyPI version License: MIT
Build Status codecov codebeat badge Codacy Badge
Installing pip install aioscrapy_redis Usage Plain text scraping
from aioscrapy_redis.core.spider import Spider
from aioscrapy_redis.https.request import Request
import re
from urllib.parse import unquote
""" The start url can be placed in start_urls or written to the redis queue """
class Async_Spider(Spider):
name = 'aioscrapy_spider'
redis_key = 'aioscrapy_spider:url'
start_urls = []
def parse(self, response):
pass
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aioscrapy_redis-0.4.tar.gz
(34.0 kB
view hashes)
Built Distribution
Close
Hashes for aioscrapy_redis-0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb5684022501fee957eaf3a58289b2f8f80923acfaec3bf4d418ae1a715be641 |
|
MD5 | 94db1094d1bb493f50908e9189c05af7 |
|
BLAKE2b-256 | 4d1dd8d673ec4541a4b34f90968f32235994620b1d44c4224b5aa0f7d1cc1983 |