s3streamer 2024.1.0.0

Stream files to AWS S3 using multipart upload.

Overview

A frontend module to upload files to AWS S3 storage. The module supports large files as it chunks them into smaller sizes and recombines them into the original file in the specified S3 bucket. It employs multiprocessing, and there is the option of specifying the size of each chunk as well as how many chunks to send in a single run. The defaults are listed in Optional Arguments below.

Prerequisites

• An AWS S3 bucket to receive uploads.

• An AWS Lambda function to perform backend tasks.

• The AWS CloudFormation template to create these resources are available in the project’s GitHub repository.

Required (Positional) Arguments

• Position 1: Filename (local full / relative path to the file)

Optional (Keyword) Arguments

• path: Destination path in the S3 bucket (default: /)

• parts: Number of multiprocessing parts to send simultaneously (default: 5)

• partsize: Size of each part in MB (default: 100)

• tmp: Location of local temporary directory to store temporary files created by the module (default: ‘/tmp’)

• purge: Whether to purge the specified file instead of uploading it (default: False)

• requrl: The endpoint URL for backend Lambda function (default: ‘https://<api_endpoint_url>’)

• reqapikey: The API key for backend Lambda function (default: ‘<api_key>’)

Usage

Installation:

pip3 install s3streamer
# or
python3 -m pip install s3streamer

In Python3:

# To upload a new file.
from s3streamer.s3streamer import multipart
response = multipart(
    'myfile.iso',
    'installer/images'
)

# To remove a file from S3.
from s3streamer.s3streamer import multipart
response = multipart(
    'myfile.iso',
    'installer/images',
    purge = True
)

In BASH:

# To upload a new file.
python3 -c \
"from s3streamer.s3streamer import multipart; \
response = multipart(\
'myfile.iso', \
'installer/images')"

# To remove a file from S3.
python3 -c \
"from s3streamer.s3streamer import multipart; \
response = multipart(\
'myfile.iso', \
'installer/images', \
purge = True)"

If the upload is successful, the file will be available at installer/images/myfile.iso.