A token bucket implementation for celery rate limiting in Django
Project description
Django celery token bucket
A dynamic token bucket implementation using the database scheduler django celery beat.
How it works
The bucket is represented by a celery queue that will not be processed by a worker but just hold our tokens (messages).
Whenever a rate limited task should be run, the decorator tries to consume a message from that queue. If the queue is empty, the task gets retried after the defined timeout.
A periodic task will then refill the bucket with tokens whenever they should be available again.
Define a token bucket
Buckets are defined in the Django config.
Following example allows one thousand tokens per hour to throttle access to a rate limited third party API.
Add to settings.py
of your project.
from typing import Dict
from celery import schedules
from django_celery_token_bucket import TokenBucket
INSTALLED_APPS = [
...,
'django_celery_token_bucket'
]
CELERY_TOKEN_BUCKETS: Dict[str, TokenBucket] = {
"my_api_client": TokenBucket(
name="my_api_client",
schedule=schedules.crontab(minute=0), # once every hour
amount=1000,
maximum=1000,
)
}
name
The name must only consist of letters, numbers and the underscore character as it's used in the name of the celery queue. It should also be the same as the key in the CELERY_TOKEN_BUCKETS dictionary.
schedule
A celery.schedules.crontab
that defines when the tokens should be refilled.
amount
The amount of tokens to add whenever the scheduled refill is run.
maximum
The maximum amount of tokens our bucket can hold.
token_refill_queue
Override this setting if you want token refill tasks for this bucket to be placed on a specific queue.
This field is by default None. If no value is provided, the CELERY_DEFAULT_QUEUE
setting is used or celery
Sync period tasks to refill the buckets
A management command token_bucket_register_periodic_tasks
is provided that should be run during deployment of your
application to sync the period tasks and make sure that buckets get properly refilled.
Use the rate_limit decorator
The decorator will make sure that the task that gets decorated will not exceed the limit of available tokens.
from my_app.celery import celery_app
from django_celery_token_bucket.decorators import rate_limit
@celery_app.task
@rate_limit(token_bucket="my_api_client", retry_backoff=300)
def my_tasK(*args, **kwargs):
return
The above task will try to consume a token from the my_api_client
and retries after 300 seconds if no token is
available. A failed token retrieval will not increase the retry count.
affect_task_retries
Defaults to False
@celery_app.task(max_retries=3, retry_backoff=60)
@rate_limit(token_bucket="my_api_client", retry_backoff=300, affect_task_retries=True)
def my_tasK(*args, **kwargs):
return
In this scenario, a failed token retrieval will increase the retry count of the task decorator. If we cannot get a token on the first try, we will start over again with the 2nd try.
Set affect_task_retries
to False
to retry token retrieval forever and only fail on max retries when the actual task execution fails.
Run the tests locally
A docker-compose environment is provided to easily run the tests:
docker-compose run --rm django test
Making a new release
bumpversion is used to manage releases.
Add your changes to the CHANGELOG, run
bumpversion <major|minor|patch>
and push (including tags).
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django-celery-token-bucket-1.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab6624928b2fde09a0560d47b632e0d7375bedcf525e1252155534dfdf64b340 |
|
MD5 | 7cbd6f46e80106e8a79a240d7302a4f0 |
|
BLAKE2b-256 | 5220574debd1ef6a0efeb3a759b98d7a857297748a2d01db3a17fc6336b9098e |
Hashes for django_celery_token_bucket-1.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d680620daa3c6ea508a8ac857ee1c0c368581dc3e0b8c6f245bc799c90ddc23 |
|
MD5 | 4f5606f6350bd28b7124003ffd5d1e28 |
|
BLAKE2b-256 | a21e7c71b46e2a665693a418f0268637af4bb8b45f7ee4fc38ff11d7f90eaac7 |