Decorate your AWS Boto3 Calls with AWSRetry.backoff(). This will allows your calls to get around the AWS Eventual Consistency Errors.
Project description
AWSRetry is a Python Decorator that can be used to wrap boto3 function calls. This function was built out of the need to get around a couple of common issues when working with AWS API’s.
- Query API Request Rate
- Eventual Consistency Model.
Exceptions that will get retried when encountered
- RequestLimitExceeded
- Unavailable
- ServiceUnavailable
- InternalFailure
- InternalError
- ^w+.NotFound
This list can be extended. (http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html)
Quick Start
Install awsretry.
$ pip install awsretry
I will assume you know about setting up Boto3 Credentials, if not you can read the instructions here http://boto3.readthedocs.io/en/latest/guide/configuration.html
Keyword Arguments that AWSRetry.backoff accepts
- tries = The number of times to try before giving up. Default = 10
- delay = The initial delay between retries in seconds. Default = 3
- backoff = backoff multiplier e.g. value of 2 will double the delay each retry. Default = 1.1
- added_exceptions = Other exceptions to retry on, beyond the defaults. Default = list()
Examples
Write a quick function that implements AWSRetry.backoff()
#!/usr/bin/env python import botocore import boto3 from awsretry import AWSRetry @AWSRetry.backoff() def get_instances(): client = boto3.client('ec2') try: instances = client.describe_instances() return instances except botocore.exceptions.ClientError as e: raise e instances = get_instances()
Write a quick function that will overwrite the default arguments.
#!/usr/bin/env python import botocore import boto3 from awsretry import AWSRetry @AWSRetry.backoff(tries=20, delay=2, backoff=1.5, added_exceptions=['ConcurrentTagAccess']) def create_tags(): client = boto3.client('ec2') try: resources = ['1-12345678891234'] tags = [{'Key': 'service', 'Value': 'web-app'}] instances = client.create_tags(Resources=resources, Tags=tags) except botocore.exceptions.ClientError as e: raise e create_tags()
Development
Assuming that you have Python and virtualenv installed, set up your environment and install the required dependencies like this instead of the pip install awsretry defined above:
$ git clone https://github.com/linuxdynasty/awsretry.git
$ cd awsretry
$ virtualenv venv
...
$ . venv/bin/activate
$ pip install -r requirements.txt
$ pip install -e .
Running Tests
You can run the tests by using tox which implements nosetest or run them directly using nosetest.
$ tox $ tox tests/test_awsretry.py $ tox -e py27,py36 tests/ $ nosetest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size gd-awsretry-1.1.2.tar.gz (4.5 kB) | File type Source | Python version None | Upload date | Hashes View |