aioscraper 0.7.0

Async framework for building modular and scalable web scrapers.

aioscraper

Asynchronous framework for building modular and scalable web scrapers.

Features

• 🚀 Fully asynchronous architecture powered by aiohttp and aiojobs
• 🔧 Modular system with middleware support
• 📦 Pipeline data processing
• ⚙️ Flexible configuration
• 🔄 Priority-based request queue management
• 🛡️ Built-in error handling

Installation

pip install aioscraper

Requirements

• Python 3.10 or higher
• aiohttp
• aiojobs

Quick Start

import asyncio

from aioscraper import AIOScraper
from aioscraper.types import Request, SendRequest, Response


async def scraper(send_request: SendRequest) -> None:
    await send_request(Request(url="https://example.com", callback=handle_response))


async def handle_response(response: Response) -> None: ...


async def main():
    async with AIOScraper(scraper) as s:
        await s.start()


if __name__ == "__main__":
    asyncio.run(main())

License

MIT License

Copyright (c) 2025 darkstussy

Links

• PyPI
• GitHub
• Issues