Skip to main content

Function decoration for backoff and retry

Project description

backoff

https://travis-ci.org/litl/backoff.svg?branch=master https://coveralls.io/repos/litl/backoff/badge.svg?branch=master https://img.shields.io/pypi/v/backoff.svg

Function decoration for backoff and retry

This module provides function decorators which can be used to wrap a function such that it will be retried until some condition is met. It is meant to be of use when accessing unreliable resources with the potential for intermittent failures i.e. network resources and external APIs. Somewhat more generally, it may also be of use for dynamically polling resources for externally generated content.

Decorators support both regular functions for synchronous code and asyncio’s coroutines for asynchronous code.

Examples

Since Kenneth Reitz’s requests module has become a defacto standard for synchronous HTTP clients in Python, networking examples below are written using it, but it is in no way required by the backoff module.

@backoff.on_exception

The on_exception decorator is used to retry when a specified exception is raised. Here’s an example using exponential backoff when any requests exception is raised:

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException)
def get_url(url):
    return requests.get(url)

The decorator will also accept a tuple of exceptions for cases where you want the same backoff behavior for more than one exception type:

@backoff.on_exception(backoff.expo,
                      (requests.exceptions.Timeout,
                       requests.exceptions.ConnectionError))
def get_url(url):
    return requests.get(url)

Give Up Conditions

Optional keyword arguments can specify conditions under which to give up.

The keyword argument max_time specifies the maximum amount of total time in seconds that can elapse before giving up.

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException,
                      max_time=60)
def get_url(url):
    return requests.get(url)

Keyword argument max_tries specifies the maximum number of calls to make to the target function before giving up.

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException,
                      max_tries=8,
                      jitter=None)
def get_url(url):
    return requests.get(url)

In some cases the raised exception instance itself may need to be inspected in order to determine if it is a retryable condition. The giveup keyword arg can be used to specify a function which accepts the exception and returns a truthy value if the exception should not be retried:

def fatal_code(e):
    return 400 <= e.response.status_code < 500

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException,
                      max_time=300,
                      giveup=fatal_code)
def get_url(url):
    return requests.get(url)

When a give up event occurs, the exception in question is reraised and so code calling an on_exception-decorated function may still need to do exception handling.

@backoff.on_predicate

The on_predicate decorator is used to retry when a particular condition is true of the return value of the target function. This may be useful when polling a resource for externally generated content.

Here’s an example which uses a fibonacci sequence backoff when the return value of the target function is the empty list:

@backoff.on_predicate(backoff.fibo, lambda x: x == [], max_value=13)
def poll_for_messages(queue):
    return queue.get()

Extra keyword arguments are passed when initializing the wait generator, so the max_value param above is passed as a keyword arg when initializing the fibo generator.

When not specified, the predicate param defaults to the falsey test, so the above can more concisely be written:

@backoff.on_predicate(backoff.fibo, max_value=13)
def poll_for_message(queue)
    return queue.get()

More simply, a function which continues polling every second until it gets a non-falsey result could be defined like like this:

@backoff.on_predicate(backoff.constant, interval=1)
def poll_for_message(queue)
    return queue.get()

Jitter

A jitter algorithm can be supplied with the jitter keyword arg to either of the backoff decorators. This argument should be a function accepting the original unadulterated backoff value and returning it’s jittered counterpart.

As of version 1.2, the default jitter function backoff.full_jitter implements the ‘Full Jitter’ algorithm as defined in the AWS Architecture Blog’s Exponential Backoff And Jitter post. Note that with this algorithm, the time yielded by the wait generator is actually the maximum amount of time to wait.

Previous versions of backoff defaulted to adding some random number of milliseconds (up to 1s) to the raw sleep value. If desired, this behavior is now available as backoff.random_jitter.

Using multiple decorators

The backoff decorators may also be combined to specify different backoff behavior for different cases:

@backoff.on_predicate(backoff.fibo, max_value=13)
@backoff.on_exception(backoff.expo,
                      requests.exceptions.HTTPError,
                      max_time=60)
@backoff.on_exception(backoff.expo,
                      requests.exceptions.TimeoutError,
                      max_time=300)
def poll_for_message(queue):
    return queue.get()

Runtime Configuration

The decorator functions on_exception and on_predicate are generally evaluated at import time. This is fine when the keyword args are passed as constant values, but suppose we want to consult a dictionary with configuration options that only become available at runtime. The relevant values are not available at import time. Instead, decorator functions can be passed callables which are evaluated at runtime to obtain the value:

def lookup_max_time():
    # pretend we have a global reference to 'app' here
    # and that it has a dictionary-like 'config' property
    return app.config["BACKOFF_MAX_TIME"]

@backoff.on_exception(backoff.expo,
                      ValueError,
                      max_time=lookup_max_time)

Event handlers

Both backoff decorators optionally accept event handler functions using the keyword arguments on_success, on_backoff, and on_giveup. This may be useful in reporting statistics or performing other custom logging.

Handlers must be callables with a unary signature accepting a dict argument. This dict contains the details of the invocation. Valid keys include:

  • target: reference to the function or method being invoked

  • args: positional arguments to func

  • kwargs: keyword arguments to func

  • tries: number of invocation tries so far

  • elapsed: elapsed time in seconds so far

  • wait: seconds to wait (on_backoff handler only)

  • value: value triggering backoff (on_predicate decorator only)

A handler which prints the details of the backoff event could be implemented like so:

def backoff_hdlr(details):
    print ("Backing off {wait:0.1f} seconds afters {tries} tries "
           "calling function {target} with args {args} and kwargs "
           "{kwargs}".format(**details))

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException,
                      on_backoff=backoff_hdlr)
def get_url(url):
    return requests.get(url)

Multiple handlers per event type

In all cases, iterables of handler functions are also accepted, which are called in turn. For example, you might provide a simple list of handler functions as the value of the on_backoff keyword arg:

@backoff.on_exception(backoff.expo,
                      requests.exceptions.RequestException,
                      on_backoff=[backoff_hdlr1, backoff_hdlr2])
def get_url(url):
    return requests.get(url)

Getting exception info

In the case of the on_exception decorator, all on_backoff and on_giveup handlers are called from within the except block for the exception being handled. Therefore exception info is available to the handler functions via the python standard library, specifically sys.exc_info() or the traceback module.

Asynchronous code

Backoff supports asynchronous execution in Python 3.5 and above.

To use backoff in asynchronous code based on asyncio you simply need to apply backoff.on_exception or backoff.on_predicate to coroutines. You can also use coroutines for the on_success, on_backoff, and on_giveup event handlers, with the interface otherwise being identical.

The following examples use aiohttp asynchronous HTTP client/server library.

@backoff.on_exception(backoff.expo, aiohttp.ClientError, max_time=60)
async def get_url(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

Logging configuration

Errors and backoff and retry attempts are logged to the ‘backoff’ logger. By default, this logger is configured with a NullHandler, so there will be nothing output unless you configure a handler. Programmatically, this might be accomplished with something as simple as:

logging.getLogger('backoff').addHandler(logging.StreamHandler())

The default logging level is INFO, which corresponds to logging anytime a retry event occurs. If you would instead like to log only when a giveup event occurs, set the logger level to ERROR.

logging.getLogger('backoff').setLevel(logging.ERROR)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

backoff-1.7.1.tar.gz (13.1 kB view details)

Uploaded Source

Built Distribution

backoff-1.7.1-py2.py3-none-any.whl (42.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file backoff-1.7.1.tar.gz.

File metadata

  • Download URL: backoff-1.7.1.tar.gz
  • Upload date:
  • Size: 13.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/0.12.10 CPython/3.7.1 Darwin/16.7.0

File hashes

Hashes for backoff-1.7.1.tar.gz
Algorithm Hash digest
SHA256 dde694bf8df7478956e9ebc1fa1de48f04ba22de6798cb5e1a237320336fc0a8
MD5 feda1941c307890ea2dab92448881f2a
BLAKE2b-256 841d1b7c1502d91f86c3bd38f693c9c037bf45d2684eaf071426da27a8a2761c

See more details on using hashes here.

File details

Details for the file backoff-1.7.1-py2.py3-none-any.whl.

File metadata

  • Download URL: backoff-1.7.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 42.3 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/0.12.10 CPython/3.7.1 Darwin/16.7.0

File hashes

Hashes for backoff-1.7.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 a2d9e32819829be681e2a1264d55172e7ddbb409b93d7388b442b894f630ba33
MD5 85c6cec669653c68cd8ac6eed7511564
BLAKE2b-256 97c296953dbfea5b46370edfcacb2ec2e02463061e9aff7370d0978bc25013ba

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page