Skip to main content

A simple and generic rate limiter to prevent too many function calls or http requests.

Project description

mThrottle

A simple and generic rate limiter to prevent too many function calls or http requests.

When making large number of HTTP requests, its essential to respect the API limits of a server.

The Throttle class ensures you do not exceed these limits.

Python version: >= 3.7

Install with PIP

pip install mthrottle

Usage

Throttle takes two arguments - a dictionary configuration and an integer maxPenaltyCount.

The max_penalty_count is the maximum number of 429 HTTP error code, allowed to be returned during a session. Exceeding this limit will result in a runtime error.

The configuration is a dictionary which defines the limits for each endpoint. Here, RPS stands for Requests Per Second and RPM stands for Requests Per Minute. Limits are set for each end point.

Configuration must have a default key which defines the default limits.

To rate limit, run th.check method before a request or function call.

from mthrottle import Throttle

config = {
    'default': {
        'rps': 3,
        'rpm': 150
    }
    'search': {
        'rps': 20
        'rpm': 500
    }
}

maxPenaltyCount = 10

th = Throttle(config, maxPenaltyCount)

for i in range(200):
    th.check()
    print(i, flush=True, end='\r' * 3)

th.check takes a key argument, matching one of the key values in config. If key is not specified, default is used.

Example using Throttle.penalize

Throttle.penalize returns a boolean and must be called when a 429 HTTP status code is returned. It returns True when maxPenaltyCount is exceeded.

from mthrottle import Throttle
import requests

config = {
    'run': {
        'rpm': 3
    }
}

maxPenaltyCount = 10

th = Throttle(config, maxPenaltyCount)

for i in range(200):
    th.check(key='run')
    r = requests.get('https://google.com')

    if r.status_code == 429:
        if th.penalize():
            raise RuntimeError('Too many API rate limit warnings.')

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mthrottle-0.0.1.tar.gz (16.0 kB view hashes)

Uploaded Source

Built Distribution

mthrottle-0.0.1-py3-none-any.whl (15.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page