Rate-limiting for the requests library
Project description
Requests-Ratelimiter
This package is a simple wrapper around pyrate-limiter that adds convenient integration with the requests library.
Full project documentation can be found at requests-ratelimiter.readthedocs.io.
Features
pyrate-limiter
is a general-purpose rate limiting library that implements the leaky bucket algorithm, supports multiple rate limits, and has optional persistence with SQLite and Redis backendsrequests-ratelimiter
adds some extra conveniences specific to sending HTTP requests with therequests
library- It can be used as a
transport adapter,
session,
or session mixin for compatibility with other
requests
-based libraries - Rate limits are tracked separately per host
- Different rate limits can optionally be applied to different hosts
Installation
pip install requests-ratelimiter
Usage
Sessions
Example with LimiterSession
:
from requests import Session
from requests_ratelimiter import LimiterSession
# Apply a rate-limit (5 requests per second) to all requests
session = LimiterSession(per_second=5)
# Make rate-limited requests that stay within 5 requests per second
for _ in range(10):
response = session.get('https://httpbin.org/get')
print(response.json())
Adapters
Example with LimiterAdapter
:
from requests import Session
from requests_ratelimiter import LimiterAdapter
session = Session()
# Apply a rate-limit (5 requests per second) to all requests
adapter = LimiterAdapter(per_second=5)
session.mount('http://', adapter)
session.mount('https://', adapter)
# Make rate-limited requests
for user_id in range(100):
response = session.get(f'https://api.some_site.com/v1/users/{user_id}')
print(response.json())
Per-Host Rate Limit Tracking
With either LimiterSession
or LimiterAdapter
, rate limits are tracked separately
for each host. In other words, requests sent to one host will not count against the rate limit for
any other hosts:
session = LimiterSession(per_second=5)
# Make requests for two different hosts
for _ in range(10):
response = session.get(f'https://httpbin.org/get')
print(response.json())
session.get(f'https://httpbingo.org/get')
print(response.json())
If you have a case where multiple hosts share the same rate limit, you can disable this behavior
with the per_host
option:
session = LimiterSession(per_second=5, per_host=False)
Per-Host Rate Limit Definitions
With LimiterAdapter
, you can apply different rate limits to different hosts or URLs:
# Apply different rate limits (2/second and 100/minute) to a specific host
adapter_2 = LimiterAdapter(per_second=2, per_minute=100)
session.mount('https://api.some_site.com', adapter_2)
Behavior for matching requests is the same as other transport adapters: requests
will use the
adapter with the most specific (i.e., longest) URL prefix for a given request. For example:
session.mount('https://api.some_site.com/v1', adapter_3)
session.mount('https://api.some_site.com/v1/users', adapter_4)
# This request will use adapter_3
session.get('https://api.some_site.com/v1/')
# This request will use adapter_4
session.get('https://api.some_site.com/v1/users/1234')
Server-Side Rate Limit Behavior
Sometimes, server-side rate limiting may not behave exactly as documented (or may not be documented
at all). Or you might encounter other scenarios where your client-side limit gets out of sync with
the server-side limit. In most cases, a server will send a 429: Too Many Requests
response for an
exceeded rate limit.
When this happens, requests-ratelimiter
will attempt to catch up to the server-side limit by
adding an extra delay before the next request. This will use the smallest rate limit interval you've
defined. For example, if you have a per-minute and per-hour limit, up to 1 minute of delay time will
be added before the next request.
If a server sends a different status code to indicate an exceeded limit, you can set this via limit_statuses
:
session = LimiterSession(per_second=5, limit_statuses=[429, 500])
Or if you would prefer to disable this behavior and handle it yourself:
session = LimiterSession(per_second=5, limit_statuses=[])
Backends
By default, rate limits are tracked in memory and are not persistent. You can optionally use either SQLite or Redis to persist rate limits across threads, processes, and/or application restarts. See pyrate-limiter docs for more details.
Compatibility
There are many other useful libraries out there that add features to requests
, most commonly by
extending or modifying
requests.Session or
requests.HTTPAdapter.
To use requests-ratelimiter
with one of these libraries, you have a few different options:
- If the library provides a custom
Session
class, mount aLimiterAdapter
on it - Or use
LimiterMixin
to create a customSession
class with features from both libraries - If the library provides a custom
Adapter
class, useLimiterMixin
to create a customAdapter
class with features from both libraries
Custom Session Example: Requests-Cache
For example, to combine with requests-cache, which also includes a separate mixin class:
from requests import Session
from requests_cache import CacheMixin, RedisCache
from requests_ratelimiter import LimiterMixin, RedisBucket
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
"""Session class with caching and rate-limiting behavior. Accepts arguments for both
LimiterSession and CachedSession.
"""
# Optionally use Redis as both the bucket backend and the cache backend
session = CachedLimiterSession(
per_second=5,
bucket_class=RedisBucket,
backend=RedisCache(),
)
This example has an extra benefit: cache hits won't count against your rate limit!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for requests-ratelimiter-0.3.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 242363e9a37fd26251bce43983554ed1b600f66b9a4eacd8819bbbe96e3bd338 |
|
MD5 | 15bdd9e3d4040ae4c70f540b7b3d9481 |
|
BLAKE2b-256 | b59bd5fed78588fa36d1978743e5a6487fe7142ef6e5371640e287589b6afe49 |
Hashes for requests_ratelimiter-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8fd1145bf89a1ab72aa568d7d324eeb76a0d9c2cca60a69edb7551be72ef048 |
|
MD5 | 06cc28bd645d2c38e9de0cf835b45769 |
|
BLAKE2b-256 | 124a28bdd8aa29e429203a7459fc4e3803e94ade1ef9d509ee97a17c0b30f785 |