AioScrapy

AioScrapy是一个基于Python异步IO的强大网络爬虫框架。它的设计理念源自Scrapy，但完全基于异步IO实现，提供更高的性能和更灵活的配置选项。
AioScrapy is a powerful asynchronous web crawling framework built on Python's asyncio library. It is inspired by Scrapy but completely reimplemented with asynchronous IO, offering higher performance and more flexible configuration options.

特性 | Features

完全异步：基于Python的asyncio库，实现高效的并发爬取
多种下载处理程序：支持多种HTTP客户端，包括aiohttp、httpx、requests、pyhttpx、curl_cffi、DrissionPage、playwright和sbcdp
灵活的中间件系统：轻松添加自定义功能和处理逻辑
强大的数据处理管道：支持多种数据库存储选项
内置信号系统：方便的事件处理机制
丰富的配置选项：高度可定制的爬虫行为
分布式爬取：支持使用Redis和RabbitMQ进行分布式爬取
数据库集成：内置支持Redis、MySQL、MongoDB、PostgreSQL和RabbitMQ
Fully Asynchronous: Built on Python's asyncio for efficient concurrent crawling
Multiple Download Handlers: Support for various HTTP clients including aiohttp, httpx, requests, pyhttpx, curl_cffi, DrissionPage, playwright and sbcdp
Flexible Middleware System: Easily add custom functionality and processing logic
Powerful Data Processing Pipelines: Support for various database storage options
Built-in Signal System: Convenient event handling mechanism
Rich Configuration Options: Highly customizable crawler behavior
Distributed Crawling: Support for distributed crawling using Redis and RabbitMQ
Database Integration: Built-in support for Redis, MySQL, MongoDB, PostgreSQL, and RabbitMQ

安装 | Installation

要求 | Requirements

Python 3.9+

使用pip安装 | Install with pip

pip install aio-scrapy

# Install the latest aio-scrapy
# pip install git+https://github.com/ConlinH/aio-scrapy

开始 | Start

from aioscrapy import Spider, logger


class MyspiderSpider(Spider):
    name = 'myspider'
    custom_settings = {
        "CLOSE_SPIDER_ON_IDLE": True
    }
    start_urls = ["https://quotes.toscrape.com"]

    @staticmethod
    async def process_request(request, spider):
        """ request middleware """
        pass

    @staticmethod
    async def process_response(request, response, spider):
        """ response middleware """
        return response

    @staticmethod
    async def process_exception(request, exception, spider):
        """ exception middleware """
        pass

    async def parse(self, response):
        for quote in response.css('div.quote'):
            item = {
                'author': quote.xpath('span/small/text()').get(),
                'text': quote.css('span.text::text').get(),
            }
            yield item

    async def process_item(self, item):
        logger.info(item)


if __name__ == '__main__':
    MyspiderSpider.start()

文档 | Documentation

文档目录 | Documentation Contents

许可证 | License

本项目采用MIT许可证 - 详情请查看LICENSE文件。
This project is licensed under the MIT License - see the LICENSE file for details.

联系

QQ: 995018884
WeChat: h995018884

Metadata

Release files for aio-scrapy 2.1.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aio-scrapy 2.1.9
File	Size	Uploaded	Provenance
aio_scrapy-2.1.9.tar.gz	299.3 kB	Feb 5, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for aio-scrapy 2.1.9
File	Interpreter	ABI	Platform	Reset
aio_scrapy-2.1.9-py3-none-any.whl 400.9 kB Feb 5, 2026 Without provenance	Python 3	none	any	Details

Total release size: 700.2 kB

Release files / aio_scrapy-2.1.9.tar.gz

Download URL	aio_scrapy-2.1.9.tar.gz
Size	299.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`a6811e9ea0b9349b1a5e29bf06fe2d04345a6cbc9bd76a136087e8b091da8921`
BLAKE2b-256 checksum How to use checksums	`1f39be43b8da441fd0cd9c19f995e6634b8c8481d7f1faa13e963b7b33295b3e`
Upload date	Feb 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.9`

Release files / aio_scrapy-2.1.9-py3-none-any.whl

Download URL	aio_scrapy-2.1.9-py3-none-any.whl
Size	400.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`1355921f7d03089bf44d5a3870fb415ae14c35412db03a4538f0044fec07c160`
BLAKE2b-256 checksum How to use checksums	`67263d62ad5a36a9c75ce1268d1710cf9357672ca4fcc51fd606faacb3490b99`
Upload date	Feb 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.11.9`

Security for aio-scrapy 2.1.9

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.