PyS3Uploader 0.4.0a1

Python module to upload objects to an S3 bucket.

Versions Supported

Language Stats

Repo Stats

Activity

Build Status

PyS3Uploader

Python module to upload an entire directory to an S3 bucket.

Bucket Policy Required

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucketsForExistenceCheck",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        },
        {
            "Sid": "ListAndUploadToSpecificBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::bucketname"
        },
        {
            "Sid": "UploadObjectsToBucket",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::bucketname/*"
        }
    ]
}

Installation

pip install PyS3Uploader

Usage

Upload objects in parallel

import pys3uploader

if __name__ == '__main__':
    wrapper = pys3uploader.Uploader(
        bucket_name="BUCKET_NAME",
        upload_dir="FULL_PATH_TO_UPLOAD",
        exclude_prefix="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
    )
    wrapper.run_in_parallel()

Upload objects in sequence

import pys3uploader

if __name__ == '__main__':
    wrapper = pys3uploader.Uploader(
        bucket_name="BUCKET_NAME",
        upload_dir="FULL_PATH_TO_UPLOAD",
        exclude_prefix="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
    )
    wrapper.run()

Mandatory arg

• bucket_name - Name of the s3 bucket.
• upload_dir - Directory to upload.

Optional kwargs

• s3_prefix - S3 object prefix for each file. Defaults to None
• exclude_prefix - Path in upload_dir that has to be excluded in object keys. Defaults to None
• skip_dot_files - Boolean flag to skip dot files. Defaults to True
• overwrite - Boolean flag to overwrite files present in S3. Defaults to False
• file_exclusion - Sequence of files to exclude during upload. Defaults to None
• folder_exclusion - Sequence of directories to exclude during upload. Defaults to None
• logger - Bring your own custom pre-configured logger. Defaults to on-screen logging.
• log_handler - Choose between stdout vs file logging. Defaults to pys3uploader.LogHandler.stdout
• log_level - Choose the logging level. Defaults to pys3uploader.LogLevel.debug
• env_file – Path to a .env file for loading environment variables. Defaults to scanning the current directory.
  
  
• region_name - AWS region name. Defaults to the env var AWS_DEFAULT_REGION
• profile_name - AWS profile name. Defaults to the env var PROFILE_NAME
• aws_access_key_id - AWS access key ID. Defaults to the env var AWS_ACCESS_KEY_ID
• aws_secret_access_key - AWS secret access key. Defaults to the env var AWS_SECRET_ACCESS_KEY

AWS values are loaded from env vars or the default config at ~/.aws/config / ~/.aws/credentials

Coding Standards

Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8 and isort

Release Notes

Requirement

python -m pip install gitverse

Usage

gitverse-release reverse -f release_notes.rst -t 'Release Notes'

Linting

pre-commit will ensure linting, run pytest, generate runbook & release notes, and validate hyperlinks in ALL markdown files (including Wiki pages)

Requirement

pip install sphinx==5.1.1 pre-commit recommonmark

Usage

pre-commit run --all-files

Pypi Package

https://pypi.org/project/PyS3Uploader/

Runbook

https://thevickypedia.github.io/PyS3Uploader/

License & copyright

© Vignesh Rao

Licensed under the MIT License