aio-scrapy

An asyncio + aiolibs crawler imitate scrapy framework

English | 中文

Overview

aio-scrapy framework is base on opensource project Scrapy & scrapy_redis.
aio-scrapy implements compatibility with scrapyd.
aio-scrapy implements redis queue and rabbitmq queue.
aio-scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.
Distributed crawling/scraping.

Requirements

Python 3.9+
Works on Linux, Windows, macOS, BSD

Install

The quick way:

# Install the latest aio-scrapy
pip install git+https://github.com/conlin-huang/aio-scrapy

# default
pip install aio-scrapy

# Install all dependencies 
pip install aio-scrapy[all]

# When you need to use mysql/httpx/rabbitmq/mongo
pip install aio-scrapy[aiomysql,httpx,aio-pika,mongo]

Usage

create project spider:

aioscrapy startproject project_quotes

cd project_quotes
aioscrapy genspider quotes

quotes.py

from aioscrapy.spiders import Spider


class QuotesMemorySpider(Spider):
    name = 'QuotesMemorySpider'

    start_urls = ['https://quotes.toscrape.com']

    async def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'author': quote.xpath('span/small/text()').get(),
                'text': quote.css('span.text::text').get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)


if __name__ == '__main__':
    QuotesMemorySpider.start()

run the spider:

aioscrapy crawl quotes

create single script spider:

aioscrapy genspider single_quotes -t single

single_quotes.py:

from aioscrapy.spiders import Spider


class QuotesMemorySpider(Spider):
    name = 'QuotesMemorySpider'
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
        'CLOSE_SPIDER_ON_IDLE': True,
        # 'DOWNLOAD_DELAY': 3,
        # 'RANDOMIZE_DOWNLOAD_DELAY': True,
        # 'CONCURRENT_REQUESTS': 1,
        # 'LOG_LEVEL': 'INFO'
    }

    start_urls = ['https://quotes.toscrape.com']

    @staticmethod
    async def process_request(request, spider):
        """ request middleware """
        pass

    @staticmethod
    async def process_response(request, response, spider):
        """ response middleware """
        return response

    @staticmethod
    async def process_exception(request, exception, spider):
        """ exception middleware """
        pass

    async def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'author': quote.xpath('span/small/text()').get(),
                'text': quote.css('span.text::text').get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

    async def process_item(self, item):
        print(item)


if __name__ == '__main__':
    QuotesMemorySpider.start()

run the spider:

aioscrapy runspider quotes.py

more commands:

aioscrapy -h

Documentation

doc

Ready

please submit your sugguestion to owner by issue

Thanks

aiohttp

scrapy

Metadata

Release files for aio-scrapy 2.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aio-scrapy 2.0.2
File	Size	Uploaded	Provenance
aio-scrapy-2.0.2.tar.gz	94.9 kB	Oct 7, 2023	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for aio-scrapy 2.0.2
File	Interpreter	ABI	Platform	Reset
aio_scrapy-2.0.2-py3-none-any.whl 137.5 kB Oct 7, 2023 Without provenance	Python 3	none	any	Details

Total release size: 232.4 kB

Release files / aio-scrapy-2.0.2.tar.gz

Download URL	aio-scrapy-2.0.2.tar.gz
Size	94.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`20ceb0fdecaef62ed1dd9f3162591813cb3db7d92533d148ab445cb6e13d5ecc`
BLAKE2b-256 checksum How to use checksums	`d9327a46e4eb82130e416c6ffbd2dfeb4a0756f10939fe5151186542b24b0ea2`
Upload date	Oct 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.3`

Release files / aio_scrapy-2.0.2-py3-none-any.whl

Download URL	aio_scrapy-2.0.2-py3-none-any.whl
Size	137.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`efd04c2b85bb847f7638ad53c66903eee996f5c32c1f83d758d7159100ab294c`
BLAKE2b-256 checksum How to use checksums	`e47c89d7299d82c95ff1402cf0d6c4e54b8d5f255a1c92b7e294925624f1a2fa`
Upload date	Oct 7, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.3`

Security for aio-scrapy 2.0.2

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.