This release is a pre-release and may not be stable for production use.
aioscraper
An async Python framework for collecting data from APIs.
You write requests and response handlers. aioscraper queues the requests, limits concurrency, and retries failed attempts. It has no selectors and no crawling engine - parse the response however you like, with BeautifulSoup or anything else (see examples/quotes.py).
It is worth reaching for when one process has to keep hundreds or thousands of requests in flight
and you would otherwise be writing that machinery yourself. For a handful of requests, use
aiohttp or httpx directly.
Installation
The core ships without an HTTP client, so pick one:
pip install "aioscraper[aiohttp]" # or [httpx], or [httpx2] - the Pydantic fork of httpx
With one installed it is picked up automatically. Install several
("aioscraper[aiohttp,httpx]") and SESSION_HTTP_BACKEND chooses between them - otherwise
aiohttp wins. [aiohttp-speedups] is the same backend plus aiohttp's own speedups extra:
aiodns, Brotli and backports.zstd.
Quick Start
Create scraper.py:
import logging
from aioscraper import AIOScraper, Request, Response, ScheduleRequest, Pipeline
from dataclasses import dataclass
logger = logging.getLogger("github_repos")
scraper = AIOScraper()
@dataclass(slots=True)
class RepoStats:
name: str
stars: int
language: str
# registers the pipeline that handles RepoStats items
@scraper.pipeline(RepoStats)
class StatsPipeline:
def __init__(self):
self.total_stars = 0
async def put_item(self, item: RepoStats) -> RepoStats:
# runs once per extracted item: store it, queue it, validate it, or aggregate as here
self.total_stars += item.stars
logger.info("✓ %s: ⭐ %s (%s)", item.name, item.stars, item.language)
return item
async def close(self):
# runs once when the scraper stops: flush buffers, close connections, report totals
logger.info("Total stars collected: %s", self.total_stars)
# registers an entry point; schedule_request is injected by parameter name
@scraper
async def get_repos(schedule_request: ScheduleRequest):
repos = (
"django/django",
"fastapi/fastapi",
"pallets/flask",
"encode/httpx",
"aio-libs/aiohttp",
)
for repo in repos:
await schedule_request(
Request(
url=f"https://api.github.com/repos/{repo}",
callback=parse_repo, # runs on a response with a status below 400
errback=on_failure, # runs on anything else: 4xx/5xx, timeouts, connection failures
cb_kwargs={"repo": repo}, # extra arguments for both of them
headers={"Accept": "application/vnd.github+json"}, # required by the GitHub API
)
)
async def parse_repo(response: Response, pipeline: Pipeline):
# the body has to be read here: the connection is released when the callback returns
data = await response.json()
await pipeline(
RepoStats(
name=data["full_name"],
stars=data["stargazers_count"],
language=data.get("language", "Unknown"),
)
)
async def on_failure(exc: Exception, repo: str):
logger.error("%s: cannot parse response: %s", repo, exc)
Run it:
aioscraper scraper
What's happening?
@scraperregisters the entry point;@scraper.pipelineregisters a pipeline forRepoStatsschedule_request()queues a request and returns; the framework dispatches it when a slot frees up- Requests run concurrently up to the limit, so responses arrive in no particular order
parse_repohandles each response,on_failurehandles each failure that was not retriedStatsPipeline.close()runs once at the end, after every request has finished
Retries are on by default. Rate limiting is not - turn it on, along with concurrency and timeouts, through environment variables before running this against a real API. The settings that matter most at that point:
SCHEDULER_CONCURRENT_REQUESTS- how many requests are in flight at once, 64 by default.SCHEDULER_READY_QUEUE_MAX_SIZE- blocks the entrypoint once this many scheduled requests are waiting, so a producer cannot queue work faster than the scraper gets through it. Defaults to0(unlimited). Only blocks scheduling from the entrypoint.SESSION_RATE_LIMIT_INTERVALwithSESSION_RATE_LIMIT_PER_GROUP- the delay between requests, applied per group (by hostname, unlessAIOScraper(group_by=...)says otherwise).SESSION_RATE_LIMIT_ADAPTIVE_ENABLED- enables adaptive rate limiting. RequiresSESSION_RATE_LIMIT_PER_GROUP=true. Increases the delay after configured errors and reduces it after consecutive successes.
The CLI is one way to run a scraper; run_scraper() and the async context manager are the others,
so aioscraper can also live inside a service you already have - see
examples/web_service.py.
Examples
Runnable, commented scrapers live in examples/:
- quotes.py - paginated HTML scraping with BeautifulSoup
- adaptive_rate_limiting.py - adjusting the interval per host from the responses
- queue_consumer.py - a long-running worker fed by Redis Pub/Sub
- web_service.py - one scraper embedded in a FastAPI app
- third_party_config.py - building
Configwith an external loader
How it compares
Scrapy is a crawler: selectors, link following, and a large ecosystem of built-in middleware. Use it to walk a website. aioscraper targets collecting data from APIs - it parses nothing for you, and a callback can schedule further requests from what it read, as quotes.py does to follow pagination. It is plain asyncio, so it drops into an existing async app instead of owning the process.
aiohttp or httpx with your own semaphore and retry loop works fine until you need per-group
pacing, backpressure between the producer and the fetchers, backoff that respects Retry-After, and
somewhere to put the results. That is roughly the code aioscraper already contains.
Performance
Throughput is stable across CPython 3.11-3.14; the numbers and the harness are in the benchmarks.
Documentation
Full documentation at aioscraper.readthedocs.io
Changelog
See CHANGELOG.md for version history and release notes.
Contributing
Please see the Contributing guide for workflow, tooling, and review expectations.
License
MIT License
Copyright (c) 2025 darkstussy
Metadata
Release files for aioscraper 1.0.0rc1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aioscraper-1.0.0rc1.tar.gz | 920.0 kB | With provenance | Details for aioscraper-1.0.0rc1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aioscraper-1.0.0rc1-py3-none-any.whl | Python 3 | none | any | Details for aioscraper-1.0.0rc1-py3-none-any.whl |
Total release size: 1.0 MB
Release files / aioscraper-1.0.0rc1.tar.gz
| Download URL | aioscraper-1.0.0rc1.tar.gz |
|---|---|
| Size | 920.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
603d71ce75027381e34c22b027900401287305bda48f38bd388109b5e215f3d5
|
|
BLAKE2b-256 checksum How to use checksums |
d6650d6019526f8fd5e99725a5484bac8e0fc981fe449fe332fe3c3bce1f5fac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logRelease files / aioscraper-1.0.0rc1-py3-none-any.whl
| Download URL | aioscraper-1.0.0rc1-py3-none-any.whl |
|---|---|
| Size | 80.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f5a126c89d36b1d9a502a6f00a2c82a76ac738e25e07848e910cb44fff64bcae
|
|
BLAKE2b-256 checksum How to use checksums |
d5e5a64b47ae9aaf7fe853bd108e1e22febd43b58fc57e0124a94796b87b4eba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logSecurity for aioscraper 1.0.0rc1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aioscraper-1.0.0rc1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.