aio-scrapy

An asyncio + aiolibs crawler imitate scrapy framework

English | 中文

Overview

aio-scrapy framework is base on opensource project Scrapy & scrapy_redis.
aio-scrapy implements compatibility with scrapyd.
aio-scrapy implements redis queue and rabbitmq queue.
aio-scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.
Distributed crawling/scraping.

Requirements

Python 3.9+
Works on Linux, Windows, macOS, BSD

Install

The quick way:

# Install the latest aio-scrapy
pip install git+https://github.com/conlin-huang/aio-scrapy

# default
pip install aio-scrapy

# Install all dependencies 
pip install aio-scrapy[all]

# When you need to use mysql/httpx/rabbitmq/mongo
pip install aio-scrapy[aiomysql,httpx,aio-pika,mongo]

Usage

create project spider:

aioscrapy startproject project_quotes

cd project_quotes
aioscrapy genspider quotes

quotes.py

from aioscrapy.spiders import Spider


class QuotesMemorySpider(Spider):
    name = 'QuotesMemorySpider'

    start_urls = ['https://quotes.toscrape.com']

    async def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'author': quote.xpath('span/small/text()').get(),
                'text': quote.css('span.text::text').get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)


if __name__ == '__main__':
    QuotesMemorySpider.start()

run the spider:

aioscrapy crawl quotes

create single script spider:

aioscrapy genspider single_quotes -t single

single_quotes.py:

from aioscrapy.spiders import Spider


class QuotesMemorySpider(Spider):
    name = 'QuotesMemorySpider'
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
        'CLOSE_SPIDER_ON_IDLE': True,
        # 'DOWNLOAD_DELAY': 3,
        # 'RANDOMIZE_DOWNLOAD_DELAY': True,
        # 'CONCURRENT_REQUESTS': 1,
        # 'LOG_LEVEL': 'INFO'
    }

    start_urls = ['https://quotes.toscrape.com']

    @staticmethod
    async def process_request(request, spider):
        """ request middleware """
        pass

    @staticmethod
    async def process_response(request, response, spider):
        """ response middleware """
        return response

    @staticmethod
    async def process_exception(request, exception, spider):
        """ exception middleware """
        pass

    async def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'author': quote.xpath('span/small/text()').get(),
                'text': quote.css('span.text::text').get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

    async def process_item(self, item):
        print(item)


if __name__ == '__main__':
    QuotesMemorySpider.start()

run the spider:

aioscrapy runspider quotes.py

more commands:

aioscrapy -h

Documentation

doc

Ready

please submit your sugguestion to owner by issue

Thanks

aiohttp

scrapy

Release files for aio-scrapy 2.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aio-scrapy 2.0.2
File	Size	Uploaded
aio-scrapy-2.0.2.tar.gz	94.9 kB	Oct 7, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for aio-scrapy 2.0.2
File	Interpreter	ABI	Platform	Reset
aio_scrapy-2.0.2-py3-none-any.whl 137.5 kB Oct 7, 2023	Python 3	none	any	Details

Total release size: 232.4 kB

Release files / aio-scrapy-2.0.2.tar.gz

Download URL	aio-scrapy-2.0.2.tar.gz
Size	94.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`20ceb0fdecaef62ed1dd9f3162591813cb3db7d92533d148ab445cb6e13d5ecc`
BLAKE2b-256 checksum How to use checksums	`d9327a46e4eb82130e416c6ffbd2dfeb4a0756f10939fe5151186542b24b0ea2`
Upload date	Oct 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.3`

Release files / aio_scrapy-2.0.2-py3-none-any.whl

Download URL	aio_scrapy-2.0.2-py3-none-any.whl
Size	137.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`efd04c2b85bb847f7638ad53c66903eee996f5c32c1f83d758d7159100ab294c`
BLAKE2b-256 checksum How to use checksums	`e47c89d7299d82c95ff1402cf0d6c4e54b8d5f255a1c92b7e294925624f1a2fa`
Upload date	Oct 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.3`