aioscraper 0.8.0

Async framework for building modular and scalable web scrapers.

aioscraper

Asynchronous framework for building modular and scalable web scrapers.

Beta notice: APIs and behavior may change; expect sharp edges while things settle.

Key Features

• Fully asynchronous architecture powered by aiohttp and aiojobs
• Modular system with middleware support
• Pipeline data processing
• Flexible configuration
• Priority-based request queue management
• Built-in error handling

Documentation

Getting started

Install

pip install aioscraper

Create scraper.py:

from aioscraper import AIOScraper
from aioscraper.types import Request, Response, SendRequest

scraper = AIOScraper()

@scraper.register
async def scrape(send_request: SendRequest):
    await send_request(Request(url="https://example.com", callback=handle_response))


async def handle_response(response: Response):
    print(f"Fetched {response.url} with status {response.status}")

Run it

aioscraper scraper

License

MIT License

Copyright (c) 2025 darkstussy