A mini spider framework, Integrate aiohttp into scrapy
Project description
Python async library for web scraping PyPI version License: MIT
Build Status codecov codebeat badge Codacy Badge
Installing pip install aioscrapy_redis Usage Plain text scraping
from aioscrapy_redis.core.spider import Spider
from aioscrapy_redis.https.request import Request
import re
from urllib.parse import unquote
""" The start url can be placed in start_urls or written to the redis queue """
class Async_Spider(Spider):
name = 'aioscrapy_spider'
redis_key = 'aioscrapy_spider:url'
start_urls = []
def parse(self, response):
pass
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aioscrapy_redis-0.5.tar.gz
(34.3 kB
view hashes)
Built Distribution
Close
Hashes for aioscrapy_redis-0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd424f9172cf3c6e24650f5511824c6fa35cd77f47b8325023d4c215981066d4 |
|
MD5 | cdfabca9a51bfa2ae0ff01434c419851 |
|
BLAKE2b-256 | 471a2cbbfcd2b7520ac935e11ed25af48e48d3290b5429549126f29787a86fe1 |