scrapequeue 0.1.0

Polite, resumable URL queues for bounded scraping jobs.

scrapequeue

Polite, resumable URL queues for bounded Python scraping jobs.

scrapequeue is for developers who already know what URLs they want to fetch and want reliable queue management without adopting a crawler framework. You provide a fetch(url) function; scrapequeue handles URL ingestion, dedupe, rate limiting, retries, SQLite checkpointing, resume, failed URL inspection, and CSV export.

When to use it

Use scrapequeue when you have:

• hundreds to low-hundreds-of-thousands of known URLs
• a script/notebook that keeps growing retry/checkpoint code
• a need to stop/restart jobs without re-fetching successes
• a custom fetcher using httpx, requests, Playwright, or another client

When not to use it

Use something else when you need:

• link discovery / crawling / spider frontiers
• distributed queues across machines
• proxy rotation, CAPTCHA solving, or anti-bot evasion
• HTML parsing helpers baked into the queue

For those, consider Scrapy/Crawlee/Playwright directly. scrapequeue intentionally stays smaller.

Install

pip install scrapequeue

For local development:

pip install -e '.[dev]'

Five-line quickstart

from scrapequeue import Queue

queue = Queue(rate_limit="1/s", retries=3, checkpoint="jobs.sqlite")
queue.add_urls(["https://example.com/a", "https://example.com/b"])
queue.run(lambda url: {"url": url, "body": "..."})
queue.export_csv("results.csv")

Real fetch function

import httpx
from scrapequeue import Queue

queue = Queue(rate_limit="30/m", retries=3, checkpoint="jobs.sqlite")
queue.add_urls("urls.csv")  # requires a `url` column


def fetch(url: str) -> dict:
    response = httpx.get(url, timeout=10, follow_redirects=True)
    response.raise_for_status()
    return {
        "status": response.status_code,
        "content_type": response.headers.get("content-type", ""),
        "body": response.text,
    }


queue.run(fetch, progress=True)
queue.export_csv("results.csv")
queue.export_failed("failed.csv")

Input sources

queue.add_urls(["https://a.com", "https://b.com"])
queue.add_urls("urls.txt")
queue.add_urls("urls.csv")  # CSV must contain a `url` column

Duplicate URLs are ignored, so re-adding the same source is safe.

State machine

Every URL is persisted in SQLite with one of these states:

pending -> in_progress -> success
pending -> in_progress -> retrying -> pending
pending -> in_progress -> failed

On startup, stranded in_progress / retrying rows are reset to pending so interrupted jobs can resume.

Retry failed URLs

queue.export_failed("failed.csv")
queue.retry_failed()
queue.run(fetch, only_failed=True)

This lets you inspect failures, fix your fetcher/network issue, then rerun only failed URLs.

Backoff and progress

queue = Queue(backoff="exponential", backoff_seconds=0.5)
queue.run(fetch, progress=True)

Backoff modes:

• fixed
• linear default
• exponential
• jitter

Progress can also be a callback:

events = []
queue.run(fetch, progress=events.append)

CLI

scrapequeue add urls.csv --checkpoint jobs.sqlite
scrapequeue status --checkpoint jobs.sqlite
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --retries 3 --rate-limit 1/s
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --only-failed --progress
scrapequeue export results.csv --checkpoint jobs.sqlite
scrapequeue export-failed failed.csv --checkpoint jobs.sqlite
scrapequeue retry-failed --checkpoint jobs.sqlite

run accepts a fetch callable as either:

module:function
/path/to/file.py:function

Public API

Queue(...)

Queue(
    checkpoint="jobs.sqlite",
    retries=3,
    rate_limit="1/s",
    backoff_seconds=0.25,
    backoff="linear",
)

Methods

queue.add_urls(urls)
queue.run(fetch, only_failed=False, progress=None)
queue.export_csv(path)
queue.export_failed(path)
queue.retry_failed()
queue.counts()

Comparison

Tool	Best for	Difference
Scrapy	full crawlers/spiders	much larger framework; scrapequeue is bounded URL queue only
Crawlee	browser/crawling automation	scrapequeue avoids crawler abstraction and external complexity
tenacity	retrying functions	scrapequeue adds URL persistence, resume, export, and state tracking
requests-cache	HTTP caching	scrapequeue is queue/checkpoint oriented, not response caching

Examples

See:

examples/basic.py
examples/httpx_fetch.py
examples/playwright_fetch.py
examples/urls.csv

Development

pytest -q
ruff check .
ruff format --check .
python -m build
twine check dist/*

Non-goals

• HTML parsing
• browser automation abstraction
• distributed queues
• proxy rotation / anti-bot evasion
• crawling/link following

License

MIT