aioscraper 0.9.0

Async framework for building modular and scalable web scrapers.

aioscraper

Asynchronous framework for building modular and scalable web scrapers.

Beta notice: APIs and behavior may change; expect sharp edges while things settle.

Key Features

• Async-first core with pluggable HTTP backends (aiohttp/httpx) and aiojobs scheduling
• Declarative flow: requests → callbacks → pipelines, with middleware hooks
• Priority queueing plus configurable concurrency
• Small, explicit API that is easy to test and compose

Getting started

Install

pip install "aioscraper[aiohttp]"
# or use httpx as the HTTP backend
pip install "aioscraper[httpx]"

Create scraper.py:

from aioscraper import AIOScraper, Request, Response, SendRequest

scraper = AIOScraper()

@scraper
async def scrape(send_request: SendRequest):
    await send_request(Request(url="https://example.com", callback=handle_response))


async def handle_response(response: Response):
    print(f"Fetched {response.url} with status {response.status}")

Run it

aioscraper scraper

Documentation

Why aioscraper?

• Scrapy is mature but tied to Twisted and a heavier, older stack. aioscraper is plain asyncio with modern typing and explicit control flow.
• Less magic: declarative Request → callback → pipeline without opaque spider classes; each piece is a normal function or typed class, simple to test and mock.
• Light footprint: pluggable HTTP backend (aiohttp/httpx), no global settings or hidden state, no vendor lock-in.
• Built for modern workloads: high-volume API/JSON crawling, fanning out to microservice endpoints, quick data collection jobs where you want async throughput without a large framework.
• Easy to embed: runs inside existing async apps (FastAPI, workers, cron jobs) without adapting to a separate runtime.

Use cases

• Collecting data from many JSON/REST endpoints concurrently
• Fan-out calls inside microservices to hydrate/cache data
• Lightweight scraping jobs that should be easy to test and ship (no big framework overhead)
• Benchmarks show stable throughput across CPython 3.11–3.14 (see the benchmarks)

Contributing

Please see the Contributing guide for workflow, tooling, and review expectations.

License

MIT License

Copyright (c) 2025 darkstussy