Fully compatible DynamoDB result backend for Celery
Dyna-Veggie is the first fully compatible DynamoDB result backend for Celery. It supports both supports both DynamoDB in AWS and DynamoDB Local. Dyna-Veggie also supports these features out of the box:
- Fully suports Native join operation in Celery just like the Redis backend, which bring the best performance for canvas operations like chord and group.
- Uses batch get for retrieving large amount of items which provides much better performance than sequential gets.
- Automatically creates the table if not presented
- Allows customizable options for DynamoDB throttling retry and exponential backoff
- Handles DynamoDB’s consistency issue appropriately
You can install Dyna-Veggie either via the Python Package Index (PyPI) or from source.
To install using pip:
$ pip install -U dyna-veggie
To use Dyna-Veggie in your celery project is simple. All you need to do is:
- Install dyna-veggie through pip
- Add the following to your celery config file AT THE BEGINNING, and Enjoy!:
# celeryconfig.py maybe? from celery.app.backends import BACKEND_ALIASES BACKEND_ALIASES['dynamodb'] = 'dyna_veggie.DynamoDBBackend' # access_key - your aws account's access key, encoded # secret_key - your aws account's secret key, encoded # region - AWS region your DynamoDB table's in # table - Optional. DynamoDB table for storing results. Default to celery # port - Optional unless you use DynamoDBLocal # read_throughput - Optional. Default to 5 # write_throughput - Optional. Default to 5 # retry_if_throttled - Optional. Default to True # consistency_wait_time - Optional. Seconds to wait for data consistency. Default to 1 result_backend = 'dynamodb://encoded_aws_access_key_id:encoded_aws_secret_access_key@region:port/table?read_throughput=n&write_throughput=m'
access_key & secret_key
The AWS region, e.g. us-east-1 or local for DynamoDB Local. See the boto3 library documentation for definition options.
The listening port of the local DynamoDB instance, if you are using the DynamoDB Local. If you have not specified the region parameter as local, setting this parameter has no effect.
Table name to use. Default is celery. See the DynamoDB Naming Rules for information on the allowed characters and length.
read_throughput & write_throughput
The Read & Write Capacity Units for the created DynamoDB table. Default is 5 for both read and write. More details can be found in the Provisioned Throughput documentation.
Whether to retry is the throttling exception is raised. Exponential backoff is used in the retry operations. More parameters for the exponential backoff behavior will be allowed to be customized in future version. Default to True.
Seconds to wait for data to become consistent before the backend thinks the data is not available. Why do we need this? Because of network latency and read consistency. Why can’t we just set ConsistentRead to True? Because the get request has the potential to reach to DynamoDB faster than a set request reaches it, if the set and get requests are issued on different machines/processes. Thus the only way to ensure we can get item in a distributed setting is to retry. Default to 1, because typically 1 second is enough for the data to present.
Apache License, Version 2.0