Rate-limiting for the requests library
Project description
Requests-Ratelimiter
This package is a thin wrapper around pyrate-limiter that adds convenient integration with the requests library.
Project documentation can be found at requests-ratelimiter.readthedocs.io.
Features
pyrate-limiter
implements the leaky bucket algorithm, supports multiple rate limits, and an optional Redis backendrequests-ratelimiter
can be used as a transport adapter, session, or session mixin for compatibility with otherrequests
-based libraries.- Rate limits can be automatically tracked separately per host, and different rate limits can be manually applied to different hosts
Installation
pip install requests-ratelimiter
Usage
Sessions
Example with LimiterSession
:
from pyrate_limiter import Duration, RequestRate
from requests import Session
from requests_ratelimiter import LimiterSession
# Apply a rate-limit (5 requests per second) to all requests
session = LimiterSession(RequestRate(5, Duration.SECOND))
# Make rate-limited requests that stay within 5 requests per second
for _ in range(10):
response = session.get('https://httpbin.org/get')
print(response.json())
Adapters
Example with LimiterAdapter
:
from pyrate_limiter import Duration, RequestRate
from requests import Session
from requests_ratelimiter import LimiterAdapter
session = Session()
# Apply a rate-limit (5 requests per second) to all requests
adapter = LimiterAdapter(RequestRate(5, Duration.SECOND))
session.mount('http://', adapter)
session.mount('https://', adapter)
# Make rate-limited requests
for user_id in range(100):
response = session.get(f'https://api.some_site.com/v1/users/{user_id}')
print(response.json())
Per-Host Rate Limits
With LimiterAdapter
, you can apply different rate limits to different hosts or URLs:
# Apply different rate limits (2/second and 100/minute) to a specific host
adapter_2 = LimiterAdapter(
RequestRate(2, Duration.SECOND),
RequestRate(100, Duration.MINUTE),
)
session.mount('https://api.some_site.com', adapter_2)
Behavior for matching requests is the same as other transport adapters: requests
will use the
adapter with the most specific (i.e., longest) URL prefix for a given request. For example:
session.mount('https://api.some_site.com/v1', adapter_3)
session.mount('https://api.some_site.com/v1/users', adapter_4)
# This request will use adapter_3
session.get('https://api.some_site.com/v1/')
# This request will use adapter_4
session.get('https://api.some_site.com/v1/users/1234')
Per-Host Rate Limit Tracking
With either LimiterSession
or LimiterAdapter
, you can automatically track rate limits separately
for each host; in other words, requests sent to one host will not count against the rate limit for
any other hosts. This can be enabled with the per_host
option:
session = LimiterSession(RequestRate(5, Duration.SECOND), per_host=True)
# Make requests for two different hosts
for _ in range(10):
response = session.get(f'https://httpbin.org/get')
print(response.json())
session.get(f'https://httpbingo.org/get')
print(response.json())
Compatibility
There are many other useful libraries out there that add features to requests
, most commonly by
extending or modifying
requests.Session.
To use requests-ratelimiter
with one of these libraries, you have at least two options:
- Mount a
LimiterAdapter
on an instance of the library'sSession
class - Use
LimiterMixin
to create a customSession
class with features from both libraries
Requests-Cache
For example, to combine with requests-cache, which also includes a separate mixin class:
from pyrate_limiter import RedisBucket, RequestRate, Duration
from requests import Session
from requests_cache import CacheMixin, RedisCache
from requests_ratelimiter import LimiterMixin
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
"""Session class with caching and rate-limiting behavior. Accepts arguments for both
LimiterSession and CachedSession.
"""
# Optionally use Redis as both the bucket backend and the cache backend
session = CachedLimiterSession(
rates=RequestRate(5, Duration.SECOND),
bucket_class=RedisBucket,
backend=RedisCache(),
)
This example has an extra benefit: cache hits won't count against your rate limit!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for requests-ratelimiter-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b9339d9cfa86a9d4b295a122411b8eaedea50f846fced47f6403145dfa0db5e |
|
MD5 | 5cd9874e8cba039cee29b92bfd9ff34a |
|
BLAKE2b-256 | 17cb7dfae82b1d784c3852c4f26f03021ca2cf65676d689538815f1ed62066c9 |
Hashes for requests_ratelimiter-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f096ccd498f9b99f5115eeb6b81c25e59fccb0ae7744c502dd8e0191b3f5345 |
|
MD5 | fbcd344e9e2e98c3bea4b5da337bed08 |
|
BLAKE2b-256 | 830ca7d16157dcc03a1dbfd961f1993d02fd7809b4bd1daef77224527a2f0084 |