boto3-oversize 0.1.3

Transparently stores oversize SNS messages in S3 and retrieves them when receiving messages using SQS.

boto3-oversize

Messages published using Amazon SNS have a maximum size of 256KiB, which can be a limitation for certain use cases. AWS provides an extended client library for Java that transparently uploads messages exceeding this threshold to S3 and restores them when the messages are received. This Python package provides the same functionality for boto3.

Installation

1. Create an Amazon S3 bucket that will store message payloads that exceed the maximum size.
   • While the Java library deletes payloads from S3 when they are retrieved, this is not appropriate when there are multiple subscribers to the topic. Instead, apply a S3 lifecycle configuration to expire the message payloads after a reasonable length of time, e.g., 14 days.
2. Install this package, e.g., pip install boto3-oversize.
3. Define the OVERSIZE_PAYLOAD_BUCKET_NAME environment variable as the name of the bucket you created; ensure your AWS access credentials have permission to call PutObject and GetObject in the root of the bucket.

Usage

The library provides replacement implementations of the core boto3 entry points that transparently apply the necessary changes to the SNS and SQS clients, both for the low-level client and service resource. Simply reference boto3_oversize instead of boto3.

Low-level client example

import boto3_oversize

sns_client = boto3_oversize.client('sns')
response = sns_client.create_topic(Name='example-large-payload-topic')
sns_client.publish(TopicArn=response['TopicArn'], Message='example-message')

Service resource example

import boto3_oversize

sqs_client = boto3_oversize.resource('sqs')
queue = sqs_client.create_queue(QueueName='example-large-payload-queue')
messages = queue.receive_messages()

Implementation

Calls to publish messages are intercepted and the message body sized check against the limit, reduced by a small percentage to consider SNS message envelope overhead if raw message delivery is not enabled. If the message exceeds this threshold, it is uploaded to an S3 bucket and the SNS message replaced with the object ARN.

When receiving messages, the SQS client checks if the entire message body appears to be an S3 object ARN. If it is, the object is retrieved from S3 and returned to the caller as the message body.