careful

careful is a Python library for writing resilient, well-behaved HTTP clients.

Code: https://codeberg.org/jpt/careful

Docs: https://jpt.sh/projects/careful/

Call one function to enchant an httpx2.Client, making your HTTP connections more resilient and better mannered.

• Configure throttling to avoid accidental Denial-of-Service / risking getting banned.
• Retries help overcome intermittent failures on flaky sites or long crawls.
• Development caching Cache persists between runs during development, reduces redundant requests made while iterating on your crawlers & scrapers.

Example

from httpx2 import Client
from careful.httpx import make_careful_client

# the only function you need to call is make_careful_client
# this wraps your existing `httpx2.Client` with your preferred
# careful behaviors

client = make_careful_client(
    client=Client(headers={'user-agent': 'spiderman/1.0'}),

    # retries are configurable w/ exponential back off
    retry_attempts=2,
    retry_wait_seconds=5,

    # can cache to process memory, filesystem, or SQLite
    cache_storage=MemoryCache(),

    # easy-to-configure throttling
    requests_per_minute=60,
)

# methods on client are called as they always are
# configured behaviors occur without further code changes
client.get("https://example.com")

Logo licensed from Adrien Coquet via Noun Project