Skip to main content

Concat files in s3

Project description

S3 Concat is used to concatenate many small files in an s3 bucket into fewer larger files.

Install

Comming soon: :sub:`pip install s3-concat`

Usage

This installs a cli called s3-concat. Find out how to use it by running s3-concat -h.
It can also be imported and used in any other python scripts by foing import s3_concat.
More docs to follow on that.

Limitations

This uses the multipart upload of s3 and its limits are https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
TODO: give details on how it deals with files smaller then 5MB because it is handled differently.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
s3-concat-0.0.3.tar.gz (4.1 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page