AIBS Informatics AWS Lambda

This is a base package that can be used standalone with some core lambda functionality or as a dependency.

Package Overview

The package contains several classes and functions that make it easy to create strongly typed lambda functions with many nice-to-have features (serialization/deserialization, easy to add metrics, utilities to create batch sqs and dynamo db event bridge processing ). In addition to these base classes, you can also use a collection of general purpose lambda handler classes.

Base Classes and Functions

LambdaHandler

The LambdaHandler class provides a base class for creating strongly typed lambda functions with features like serialization/deserialization, logging, and metrics.

ApiLambdaHandler and ApiResolverBuilder

These classes extend the LambdaHandler class and provide a way to create strongly typed lambda functions that can be used as API Gateway endpoints.

• ApiLambdaHandler: A base class for API Gateway handlers.
• ApiResolverBuilder: A utility class for building API Gateway resolvers.

Standalone Lambda Classes in this Package

Most of these lambda functions are found under src/aibs_informatics_aws_lambda/handlers/

AWS Batch Functions

• CreateDefinitionAndPrepareArgsHandler: Handles the creation and preparation of AWS Batch job definitions.
• PrepareBatchDataSyncHandler: Prepares data synchronization tasks for AWS Batch.

Data Sync Functions

Data Sync Operations (Writing, Reading and Syncing Data)

• GetJSONFromFileHandler: Retrieves JSON data from a file.
• PutJSONToFileHandler: Writes JSON data to a file.
• DataSyncHandler: Simple data sync task.
• BatchDataSyncHandler: Handles batch of data sync tasks.
• PrepareBatchDataSyncHandler: Taking a data sync request, it analyzes and generates multiple batches of data sync tasks to evenly distribute the load across multiple batch data sync tasks.

Data Sync File System Functions (Managing Data Paths)

• GetDataPathStatsHandler: Retrieves statistics about data paths.
• ListDataPathsHandler: Lists data paths.
• OutdatedDataPathScannerHandler: Scans for outdated data paths.
• RemoveDataPathsHandler: Removes data paths.

Demand Functions

• PrepareDemandScaffoldingHandler: Prepares scaffolding for demand execution.

Notification Functions

• NotificationRouter: Routes notifications to the appropriate notifier.
• SESNotifier: Sends notifications via Amazon SES.
• SNSNotifier: Sends notifications via Amazon SNS.

ECR Image Replicator

• ImageReplicatorHandler: Handles the replication of ECR images between repositories using the ECRImageReplicator.

CLI Invocation

With this package, you can also invoke lambda functions from the command line. The CLI executable is installed as handle-lambda-request and can be used to invoke lambda functions with payloads that can be specified as JSON, files, or S3 objects.

usage: handle-lambda-request [-h] [--handler-qualified-name HANDLER_QUALIFIED_NAME] [--payload PAYLOAD] [--response-location RESPONSE_LOCATION]

CLI AWS Lambda Handler

options:
  -h, --help            show this help message and exit
  --handler-qualified-name HANDLER_QUALIFIED_NAME, --handler-name HANDLER_QUALIFIED_NAME, --handler HANDLER_QUALIFIED_NAME
                        handler function qualified name. If not provided, will try to load from ('AWS_LAMBDA_FUNCTION_HANDLER', '_HANDLER') env variables
  --payload PAYLOAD, --event PAYLOAD, -e PAYLOAD
                        event payload of function. If not provided, will try to load from AWS_LAMBDA_EVENT_PAYLOAD env variable
  --response-location RESPONSE_LOCATION, -o RESPONSE_LOCATION
                        optional response location to store response at. can be S3 or local file. If not provided, will load from AWS_LAMBDA_EVENT_RESPONSE_LOCATION env variable.

Examples

Invoking a Lambda Function with a JSON Payload

handle-lambda-request --handler-qualified-name aibs_informatics_aws_lambda.handlers.data_sync.operations.GetJSONFromFileHandler --payload '{"path": "/path/to/file.json"}' --response-location /tmp/response.json

Invoking a Lambda Function with a JSON Payload from a File

handle-lambda-request --handler-qualified-name aibs_informatics_aws_lambda.handlers.data_sync.operations.GetJSONFromFileHandler --payload-file /path/to/payload.json --response-location /tmp/response.json

Invoking a Lambda Function with a JSON Payload from S3 and Saving the Response to S3

handle-lambda-request --handler-qualified-name aibs_informatics_aws_lambda.handlers.data_sync.operations.GetJSONFromFileHandler --payload-file s3://bucket/key/payload.json --response-location s3://bucket/key/response.json

Invoking a Lambda Function with environment variables

AWS_LAMBDA_EVENT_PAYLOAD='{"path": "/path/to/file.json"}'
AWS_LAMBDA_EVENT_RESPONSE_LOCATION='/tmp/response.json'
handle-lambda-request --handler-qualified-name aibs_informatics_aws_lambda.handlers.data_sync.operations.GetJSONFromFileHandler

Testing

The package includes comprehensive tests for all handlers, which can be found under the test directory.

Contributing

Any and all PRs are welcome. Please see CONTRIBUTING.md for more information.

Licensing

This software is licensed under the Allen Institute Software License, which is the 2-clause BSD license plus a third clause that prohibits redistribution and use for commercial purposes without further permission. For more information, please visit Allen Institute Terms of Use.