A mini spider framework, Integrate aiohttp into scrapy
Project description
Python async library for web scraping PyPI version License: MIT
Build Status codecov codebeat badge Codacy Badge
Installing pip install aioscrapy_redis Usage Plain text scraping
from aioscrapy_redis.core.spider import Spider
from aioscrapy_redis.https.request import Request
import re
from urllib.parse import unquote
""" The start url can be placed in start_urls or written to the redis queue """
class Async_Spider(Spider):
name = 'aioscrapy_spider'
redis_key = 'aioscrapy_spider:url'
start_urls = []
def parse(self, response):
pass
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aioscrapy_redis-0.6.tar.gz
(34.1 kB
view hashes)
Built Distribution
Close
Hashes for aioscrapy_redis-0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c18c59e806570e806e1e11c81386e2b165387e847ce5864d090635c288447ab |
|
MD5 | 8f9edb3dba3e8bafc6fa6b0235135466 |
|
BLAKE2b-256 | a2ee43dcd23fbfde0743f46e390c324ad39172209760c63e02b1203ce5947d7d |