s3shutil 0.37

Design Principles

• A simple and intuitive string based API.

• Symmetric API: download and uploads work equally

• Exposes powerful and performant one-liners.

• Emulate the well known shutil standard module API.

• Use performance boosts behind the scenes (multithreading, batching, server to server operations)

• No dependencies except boto3

Using s3shutil

s3shutil uses boto3 internally and we assume you have your credentials set up properly.

Using s3shutil is super easy:

Import is mandatory, no suprises here:

import s3shutil

Then you can do powerful things with simple one liners::

# download a tree from s3
s3shutil.copytree('s3://bucket/my/path', '/home/myuser/files/')

# upload a tree to s3
s3shutil.copytree('/home/users/pics/', 's3://bucket/path/archive/')

# copy between two s3 locations
# same or different bucket
s3shutil.copytree('s3://bucket2/files/someth/', 's3://bucket1/backup/old/')

# delete (recursively) entire prefix
s3shutil.rmtree('s3://bucket/my-files/documents/')

Just released! (December 2023), tree_sync operation:

Only copies files that are missing in the destination. Also deletes extra files.

# sync download
s3shutil.tree_sync('s3://bucket/files/docs/', '/home/myuser/docs')

# sync upload
s3shutil.tree_sync('/home/myuser/files/', 's3://bucket/files/docs-v2/')

# sync two bucket locations
s3shutil.tree_sync('s3://bucket/files/docs/', 's3://bucket2/a/b/c')

Conclusions

s3shutil will notice alone if the location is s3 (starts with s3://) or not All operations have a similar string based API of powerfull one liners

Test Matrix

s3shutil is thoroughly unit tested in all the combinations of:

Python Versions:

• 3.12

• 3.11

• 3.10

• 3.9

• 3.8

• 3.7

And boto3 Versions:

• 1.33

• 1.30

• 1.28

• 1.27

• 1.26

• 1.25

• 1.24

• 1.23

Contact

Just use it! You can send an email as well andyworms@gmail.com. All emails are (eventually) answered. Also read the code, fork, open a PR, start a discussion.