A python module for uploading data to s3 using the s3 multipart API
A python module for uploading data to s3 using the multipart API.
Install via pip:
$ pip install s3upload
Use the s3upload CLI to upload from stdin or from data provided on the command line.
Usage: s3upload [options] Options: -h, --help show this help message and exit -b BUCKET, --bucket=BUCKET the s3 bucket to upload to -k KEY, --key=KEY the name of the key to create in the bucket -K AWS_KEY, --aws_key=AWS_KEY aws access key -s AWS_SECRET, --aws_secret=AWS_SECRET aws secret key -d DATA, --data=DATA the data to upload to s3 -- if left blank will be read from STDIN -t THREADS, --threads=THREADS number of threads to use while uploading in parallel
You can also interface with the module from python:
>>> from s3upload import upload
def upload(bucket, aws_access_key, aws_secret_key, iterable, key, progress_cb=None, threads=5, replace=False, secure=True): ''' Upload data to s3 using the s3 multipart upload API. Args: bucket: name of s3 bucket aws_access_key: aws access key aws_secret_key: aws secret key iterable: The data to upload. Each 'part' in the list will be uploaded in parallel. Each part must be at least 5242880 bytes (5mb). key: the name of the key to create in the s3 bucket progress_cb: will be called with (part_no, uploaded, total) each time a progress update is available. threads: the number of threads to use while uploading. replace: will replace the key in s3 if set to true. (Default is false) secure: use ssl when talking to s3. (Default is true) '''