eib-aws-utils

Helper used by Engie Impact Sustainability Solution EMEAI to develop services based on AWS.

This package is in alpha version

Background jobs

For lambdas and batch background job, meaning not exposed though API-Gateway, you can use the entry_point decorator to manage the error handling and to configure the logging. This decorator must be used only on the main handler. It will:

• configure the standard logging package to use JSON format (see).
• catch all exception and log them correctly.

Usage

from eib_aws_utils.backgound_job import entry_point

@entry_point("my_package_name")
def my_handler(event, context):
    pass

API-Endpoints

Lambda exposing data behind API-Gateway can use the http_endpoint decorator. This decorator manage the error handling, the logging configuration and the serialization of the output.

Error handling

There is two category of error.

• The BackendError who are internal errors. Meaning errors occurring inside the application and don't have any link with the end users. For example, missing data on an external source. These errors should be catch somewhere in the application. If they are not catch then the decorator will log the error and return a 500 - Internal Server Error to the end user.

• The ClientError are error that must be return to the end user. They will be catch by the decorator and logged as warning. The error will then be serialized and returned as a JSON. For example, if the resource didn't exist you can raise a NotFoundError.

Logging configuration

The decorator will use the configure_logging function to configure properly the logging. For the details see the logging section.

Serialization

The decorator also manage the serialisation of the lambda output to match the api-gateway format. Your function can return only the body or a tuple with the body and the http status code.

The body can be an object or a dictionary. If it's an object then it must have a to_dict() function that return a dictionary representation of the object. If it's a dictionary then it will be serialized as a JSON string. In addition to the python basic data types, the dictionary can contain the following types:

• Decimal -> float
• datetime -> ISO string
• ClientError -> dictionary
• object with to_dict function -> dictionary

Usage

from eib_aws_utils.api import http_endpoint

@http_endpoint("my_package_name")
def my_handler(event, context):
    pass

Logging configuration

This library also manage the logging configuration.

The configure_logging function will configure the standard logging package to use JSON format. In addition, it get the aws_request_id from the context to add it in every log. This allow use to track every logs that belong to the same run.

The root level of logging will be set to INFO and for the package gave in parameter it will use the level defined by the LOGGING_LEVEL environment variable.

This function must be used at the beginning of the handler if you don't use one of the decorator. Otherwise the decorator manage the configuration for you.

Usage

from eib_aws_utils.logging import configure_logging

def my_handler(event, context):
    configure_logging("my_package_name", context)

DynamoDB Utils

Float compatible resource

The dynamoDB serializer can be changed to map Decimal values to float. And the reverse. By patching the default serializer you can work with float for decimal values instead of Decimal.

For this you can patch yourself the serializer when you create your boto3 resource. Or you can use the create_resource_with_float_serializer function to get the patched version of the resource.

Usage

If you patch the dynamo resource manually when you create it:

import boto3
from unittest.mock import patch
from eib_aws_utils.dynamo_utils import FloatSerializer, FloatDeserializer

with patch("boto3.dynamodb.types.TypeSerializer", new=FloatSerializer), \
     patch("boto3.dynamodb.types.TypeDeserializer", new=FloatDeserializer):
    dynamodb = boto3.resource("dynamodb")

Or get directly the patched version:

from eib_aws_utils import dynamo_utils

dynamodb = dynamo_utils.create_resource_with_float_serializer()

Query all results page at once

The query_all_items function will iterate over all results page of the query and give you all the items. You need to provide the DynamoDB resource, the table name and then you can use all the arguments available with the query function of the DynamoDB Table.

Usage

from eib_aws_utils import dynamo_utils
from boto3.dynamodb.conditions import Key

all_customer = dynamo_utils.query_all_items(
   dynamodb_resource,
   "my-table-name",
    KeyConditionExpression=Key('partition_key').eq('123456'), ScanIndexForward=False, ...
)

Http Utils

You can create easily a requests session with the retry policy and API key authentication with the create_requests_session function. You can pass the API key in parameter. This key will be set in the x-api-key header. You can also provide other headers in the parameters, configure the exponential backoff or change the default timeout. The default timeout can always be override on each call.

Usage

from eib_aws_utils import http_utils

# The most simple session with authentication
session = http_utils.create_requests_session(api_gateway_key="MY-API-KEY")

# A more complex example
session = http_utils.create_requests_session(
  headers={"Accept": "application/json"},
  api_gateway_key="MY-API-KEY",
  max_retry=25,
  backoff_factor=1.5,
  timeout=30
)

Change logs

2.0.0

• Migration to python 3.9

1.0.5

• New backend error for invalid event: InvalidEventError

1.0.4

• http_utils: improve session
  • Add a timeout on http call
  • Add retry on connection issue

1.0.3

• Fix dynamo query_all_items

1.0.2

• Fix dynamo patching

1.0.1

• Improve function wrapping in the decorators

1.0.0

Breaking changes

• Dynamo_serializer has been renamed to dynamo_utils.
• The entry_point and http_endpoint decorator need the main_module_name parameter. And they configure the logging automatically.

New features

• Creation of float compatible DynamoDB resource trough dynamo_utils module.
• Query all result pages trough dynamo_utils module.
• Get requests session with retry policy and api-key module.
• Generate a context id if there is no aws_request_id in the logging context.

0.0.3

• Bug fixes

0.0.2

• Bug fixes