Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

aioscraper

aioscraper logo

Python GitHub License PyPI - Version PyPI - Downloads GitHub Actions Workflow Status codecov Read the Docs GitHub last commit

An async Python framework for collecting data from APIs.

You write requests and response handlers. aioscraper queues the requests, limits concurrency, and retries failed attempts. It has no selectors and no crawling engine - parse the response however you like, with BeautifulSoup or anything else (see examples/quotes.py).

It is worth reaching for when one process has to keep hundreds or thousands of requests in flight and you would otherwise be writing that machinery yourself. For a handful of requests, use aiohttp or httpx directly.

Installation

The core ships without an HTTP client, so pick one:

pip install "aioscraper[aiohttp]"  # or [httpx], or [httpx2] - the Pydantic fork of httpx

With one installed it is picked up automatically. Install several ("aioscraper[aiohttp,httpx]") and SESSION_HTTP_BACKEND chooses between them - otherwise aiohttp wins. [aiohttp-speedups] is the same backend plus aiohttp's own speedups extra: aiodns, Brotli and backports.zstd.

Quick Start

Create scraper.py:

import logging
from aioscraper import AIOScraper, Request, Response, ScheduleRequest, Pipeline
from dataclasses import dataclass

logger = logging.getLogger("github_repos")
scraper = AIOScraper()


@dataclass(slots=True)
class RepoStats:
    name: str
    stars: int
    language: str


# registers the pipeline that handles RepoStats items
@scraper.pipeline(RepoStats)
class StatsPipeline:
    def __init__(self):
        self.total_stars = 0

    async def put_item(self, item: RepoStats) -> RepoStats:
        # runs once per extracted item: store it, queue it, validate it, or aggregate as here
        self.total_stars += item.stars
        logger.info("✓ %s: ⭐ %s (%s)", item.name, item.stars, item.language)
        return item

    async def close(self):
        # runs once when the scraper stops: flush buffers, close connections, report totals
        logger.info("Total stars collected: %s", self.total_stars)


# registers an entry point; schedule_request is injected by parameter name
@scraper
async def get_repos(schedule_request: ScheduleRequest):
    repos = (
        "django/django",
        "fastapi/fastapi",
        "pallets/flask",
        "encode/httpx",
        "aio-libs/aiohttp",
    )

    for repo in repos:
        await schedule_request(
            Request(
                url=f"https://api.github.com/repos/{repo}",
                callback=parse_repo,  # runs on a response with a status below 400
                errback=on_failure,  # runs on anything else: 4xx/5xx, timeouts, connection failures
                cb_kwargs={"repo": repo},  # extra arguments for both of them
                headers={"Accept": "application/vnd.github+json"},  # required by the GitHub API
            )
        )


async def parse_repo(response: Response, pipeline: Pipeline):
    # the body has to be read here: the connection is released when the callback returns
    data = await response.json()
    await pipeline(
        RepoStats(
            name=data["full_name"],
            stars=data["stargazers_count"],
            language=data.get("language", "Unknown"),
        )
    )


async def on_failure(exc: Exception, repo: str):
    logger.error("%s: cannot parse response: %s", repo, exc)

Run it:

aioscraper scraper

What's happening?

  1. @scraper registers the entry point; @scraper.pipeline registers a pipeline for RepoStats
  2. schedule_request() queues a request and returns; the framework dispatches it when a slot frees up
  3. Requests run concurrently up to the limit, so responses arrive in no particular order
  4. parse_repo handles each response, on_failure handles each failure that was not retried
  5. StatsPipeline.close() runs once at the end, after every request has finished

Retries are on by default. Rate limiting is not - turn it on, along with concurrency and timeouts, through environment variables before running this against a real API. The settings that matter most at that point:

  • SCHEDULER_CONCURRENT_REQUESTS - how many requests are in flight at once, 64 by default.
  • SCHEDULER_READY_QUEUE_MAX_SIZE - blocks the entrypoint once this many scheduled requests are waiting, so a producer cannot queue work faster than the scraper gets through it. Defaults to 0 (unlimited). Only blocks scheduling from the entrypoint.
  • SESSION_RATE_LIMIT_INTERVAL with SESSION_RATE_LIMIT_PER_GROUP - the delay between requests, applied per group (by hostname, unless AIOScraper(group_by=...) says otherwise).
  • SESSION_RATE_LIMIT_ADAPTIVE_ENABLED - enables adaptive rate limiting. Requires SESSION_RATE_LIMIT_PER_GROUP=true. Increases the delay after configured errors and reduces it after consecutive successes.

The CLI is one way to run a scraper; run_scraper() and the async context manager are the others, so aioscraper can also live inside a service you already have - see examples/web_service.py.

Examples

Runnable, commented scrapers live in examples/:

How it compares

Scrapy is a crawler: selectors, link following, and a large ecosystem of built-in middleware. Use it to walk a website. aioscraper targets collecting data from APIs - it parses nothing for you, and a callback can schedule further requests from what it read, as quotes.py does to follow pagination. It is plain asyncio, so it drops into an existing async app instead of owning the process.

aiohttp or httpx with your own semaphore and retry loop works fine until you need per-group pacing, backpressure between the producer and the fetchers, backoff that respects Retry-After, and somewhere to put the results. That is roughly the code aioscraper already contains.

Performance

Throughput is stable across CPython 3.11-3.14; the numbers and the harness are in the benchmarks.

Documentation

Full documentation at aioscraper.readthedocs.io

Changelog

See CHANGELOG.md for version history and release notes.

Contributing

Please see the Contributing guide for workflow, tooling, and review expectations.

License

MIT License

Copyright (c) 2025 darkstussy

Metadata

Release files for aioscraper 1.0.0rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aioscraper 1.0.0rc1
File Size Uploaded Provenance Actions
aioscraper-1.0.0rc1.tar.gz 920.0 kB With provenance Details for aioscraper-1.0.0rc1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aioscraper 1.0.0rc1
File Interpreter ABI Platform Actions
aioscraper-1.0.0rc1-py3-none-any.whl Python 3 none any Details for aioscraper-1.0.0rc1-py3-none-any.whl

Total release size: 1.0 MB

Release files / aioscraper-1.0.0rc1.tar.gz

Download URL aioscraper-1.0.0rc1.tar.gz
Size 920.0 kB
Tags Source
SHA-256 checksum
How to use checksums
603d71ce75027381e34c22b027900401287305bda48f38bd388109b5e215f3d5
BLAKE2b-256 checksum
How to use checksums
d6650d6019526f8fd5e99725a5484bac8e0fc981fe449fe332fe3c3bce1f5fac
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Release files / aioscraper-1.0.0rc1-py3-none-any.whl

Download URL aioscraper-1.0.0rc1-py3-none-any.whl
Size 80.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f5a126c89d36b1d9a502a6f00a2c82a76ac738e25e07848e910cb44fff64bcae
BLAKE2b-256 checksum
How to use checksums
d5e5a64b47ae9aaf7fe853bd108e1e22febd43b58fc57e0124a94796b87b4eba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Security for aioscraper 1.0.0rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aioscraper-1.0.0rc1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page