Skip to main content

Concat files in s3

Project description

Python S3 Concat


S3 Concat is used to concatenate many small files in an s3 bucket into fewer larger files.


pip install s3-concat


Command Line

$ s3-concat -h


from s3_concat import S3Concat

path_to_concat = 'PATH_TO_FILES_TO_CONCAT'
concatenated_file = 'FILE_TO_SAVE_TO.json'
# Setting this to a size will always add a part number at the end of the file name
min_file_size = '50MB'  # ex: FILE_TO_SAVE_TO-1.json, FILE_TO_SAVE_TO-2.json, ...
# Setting this to None will concat all files into a single file
# min_file_size = None  ex: FILE_TO_SAVE_TO.json

# Init the job
job = S3Concat(bucket, concatenated_file, min_file_size,
              #  session=boto3.session.Session(),  # For custom aws session
              # s3_client_kwargs={}  # Use to pass arguments allowed by the s3 client:
# Add files, can call multiple times to add files from other directories
# Add a single file at a time
# Only use small_parts_threads if you need to. See Advanced Usage section below.

Advanced Usage

Depending on your use case, you may want to use small_parts_threads.

  • small_parts_threads is only used when the files you are trying to concat are less then 5MB. Due to the limitations of the s3 multipart_upload api (see Limitations below) any files less then 5MB need to be download locally, concated together, then re uploaded. By setting this thread count it will download the parts in parallel for faster creation of the concatination process.

The values set for these arguments depends on your use case and the system you are running this on.


This uses the multipart upload of s3 and its limits are

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for s3-concat, version 0.2.3
Filename, size File type Python version Upload date Hashes
Filename, size s3-concat-0.2.3.tar.gz (7.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page