Python module to download all the objects in an S3 bucket.
Project description
Versions Supported
Language Stats
Repo Stats
Activity
Build Status
S3 Download
Python module to download all the objects in an S3 bucket.
Install from pypi
pip install s3-downloader
Usage
Environment Variables
Environment variables can optionally be loaded from a .env
file.
These env vars may also be passed as arguments during object instantiation.
AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
Using multi-threading
from s3.dumper import Downloader
if __name__ == '__main__':
wrapper = Downloader(bucket_name='BUCKET_NAME')
wrapper.run_in_parallel(threads=10) # Defaults to 5
Without using multi-threading
from s3.dumper import Downloader
if __name__ == '__main__':
wrapper = Downloader(bucket_name='BUCKET_NAME')
wrapper.run()
Coding Standards
Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8
and
isort
Linting
PreCommit
will ensure linting, and the doc creation are run on every commit.
Requirement
pip install --no-cache --upgrade sphinx pre-commit recommonmark
Usage
pre-commit run --all-files
Pypi Package
https://pypi.org/project/s3-downloader/
Runbook
https://thevickypedia.github.io/s3-downloader/
License & copyright
© Vignesh Sivanandha Rao
Licensed under the MIT License
Change Log
0.0.6 (04/17/2022)
- Add boto3 retry logic
- Update README.md
0.0.5 (04/17/2022)
- Add OOP section for easy usage
- Update docs
0.0.4 (04/17/2022)
- Implement multi-threading for performance
0.0.3 (04/17/2022)
- Base script to download s3 objects
0.0.2 (04/17/2022)
- Setup pypi package and sphinx docs
- Add all requirements
0.0.1 (04/17/2022)
- Initial commit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for s3_downloader-0.0.7-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e88259026c71ceee66930335846a104a8e9d8f6bf0ec9b412c05c77bedc6d34 |
|
MD5 | 437029469bebbb40c2aba9df02b94f13 |
|
BLAKE2b-256 | 8c4baeba0e3e79bba2c85e5765cf77e9ea59394be69e92f4e50c3ba81c6c1cbe |