A mini spider framework, Integrate aiohttp into scrapy
Project description
Python async library for web scraping PyPI version License: MIT
Build Status codecov codebeat badge Codacy Badge
Installing pip install aioscrapy_redis Usage Plain text scraping
from aioscrapy_redis.core.spider import Spider
from aioscrapy_redis.https.request import Request
import re
from urllib.parse import unquote
""" The start url can be placed in start_urls or written to the redis queue """
class Async_Spider(Spider):
name = 'aioscrapy_spider'
redis_key = 'aioscrapy_spider:url'
start_urls = []
def parse(self, response):
pass
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aioscrapy_redis-0.3.tar.gz
(34.0 kB
view hashes)
Built Distribution
Close
Hashes for aioscrapy_redis-0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42d854b453c72b82c0008fe52d5212854d88a3c23e5ed2057fff9cec7b8b69b0 |
|
MD5 | 48b8b14fef1a4a54f0478c0662c8f330 |
|
BLAKE2b-256 | 3fcfda43eedeb7d33772ede5bb4f5d0ec4036fb8740e237d8f76c4b04b62fc8a |