requests-ratelimiter 0.3.0

Rate-limiting for the requests library

Requests-Ratelimiter

This package is a thin wrapper around pyrate-limiter that adds convenient integration with the requests library.

Project documentation can be found at requests-ratelimiter.readthedocs.io.

Features

• pyrate-limiter implements the leaky bucket algorithm, supports multiple rate limits, and optional persistence with SQLite and Redis backends
• requests-ratelimiter can be used as a transport adapter, session, or session mixin for compatibility with other requests-based libraries.
• Rate limits can be automatically tracked separately per host, and different rate limits can be manually applied to different hosts

Installation

pip install requests-ratelimiter

Usage

Sessions

Example with LimiterSession:

from requests import Session
from requests_ratelimiter import LimiterSession

# Apply a rate-limit (5 requests per second) to all requests
session = LimiterSession(per_second=5)

# Make rate-limited requests that stay within 5 requests per second
for _ in range(10):
    response = session.get('https://httpbin.org/get')
    print(response.json())

Adapters

Example with LimiterAdapter:

from requests import Session
from requests_ratelimiter import LimiterAdapter

session = Session()

# Apply a rate-limit (5 requests per second) to all requests
adapter = LimiterAdapter(per_second=5)
session.mount('http://', adapter)
session.mount('https://', adapter)

# Make rate-limited requests
for user_id in range(100):
    response = session.get(f'https://api.some_site.com/v1/users/{user_id}')
    print(response.json())

Per-Host Rate Limits

With LimiterAdapter, you can apply different rate limits to different hosts or URLs:

# Apply different rate limits (2/second and 100/minute) to a specific host
adapter_2 = LimiterAdapter(per_second=2, per_minute=100)
session.mount('https://api.some_site.com', adapter_2)

Behavior for matching requests is the same as other transport adapters: requests will use the adapter with the most specific (i.e., longest) URL prefix for a given request. For example:

session.mount('https://api.some_site.com/v1', adapter_3)
session.mount('https://api.some_site.com/v1/users', adapter_4)

# This request will use adapter_3
session.get('https://api.some_site.com/v1/')

# This request will use adapter_4
session.get('https://api.some_site.com/v1/users/1234')

Per-Host Rate Limit Tracking

With either LimiterSession or LimiterAdapter, you can automatically track rate limits separately for each host; in other words, requests sent to one host will not count against the rate limit for any other hosts. This can be enabled with the per_host option:

session = LimiterSession(per_second=5, per_host=True)

# Make requests for two different hosts
for _ in range(10):
    response = session.get(f'https://httpbin.org/get')
    print(response.json())
    session.get(f'https://httpbingo.org/get')
    print(response.json())

Server-Side Rate Limit Behavior

Sometimes, server-side rate limiting may not behave exactly as documented (or may not be documented at all). Or you might encounter other scenarios where your client-side limit gets out of sync with the server-side limit. In most cases, a server will send a 429: Too Many Requests response for an exceeded rate limit.

requests-ratelimiter will handle this by setting the local limit tracker to a "limit exceeded" state. If a server sends a different status code to indicate an exceeded limit, you can set this via limit_statuses:

session = LimiterSession(per_second=5, limit_statuses=[429, 500])

Or if you would prefer to disable this behavior and handle it yourself:

session = LimiterSession(per_second=5, limit_statuses=[])

Backends

By default, rate limits are tracked in memory and are not persistent. You can optionally use either SQLite or Redis to persist rate limits across threads, processes, and/or application restarts. See pyrate-limiter docs for more details.

Compatibility

There are many other useful libraries out there that add features to requests, most commonly by extending or modifying requests.Session or requests.HTTPAdapter.

To use requests-ratelimiter with one of these libraries, you have a few different options:

1. If the library provides a custom Session class, mount a LimiterAdapter on it
2. Or use LimiterMixin to create a custom Session class with features from both libraries
3. If the library provides a custom Adapter class, use LimiterMixin to create a custom Adapter class with features from both libraries

Custom Session Example: Requests-Cache

For example, to combine with requests-cache, which also includes a separate mixin class:

from pyrate_limiter import RedisBucket
from requests import Session
from requests_cache import CacheMixin, RedisCache
from requests_ratelimiter import LimiterMixin


class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
    """Session class with caching and rate-limiting behavior. Accepts arguments for both
    LimiterSession and CachedSession.
    """


# Optionally use Redis as both the bucket backend and the cache backend
session = CachedLimiterSession(
    per_second=5,
    bucket_class=RedisBucket,
    backend=RedisCache(),
)

This example has an extra benefit: cache hits won't count against your rate limit!