Skip to main content

Easy pythonic API to copy and sync to and from s3

Project description

Unittests License Downloads Language PyVersions

s3shutil is the easiest to use and fastest way of moving around directories and files in s3.


We recommend installing from the official PyPI repository.

$ pip install s3shutil

Design Principles

  • A simple and intuitive string based API.

  • Symmetric API: download and uploads work equally

  • Exposes powerful and performant one-liners.

  • Emulate the well known shutil standard module API.

  • Use performance boosts behind the scenes (multithreading, batching, server to server operations)

  • No dependencies except boto3

Using s3shutil

s3shutil uses boto3 internally and we assume you have your credentials set up properly.

Using s3shutil is super easy:

Import is mandatory, no suprises here:

import s3shutil

Then you can do powerful things with simple one liners::

# download a tree from s3
s3shutil.copytree('s3://bucket/my/path', '/home/myuser/files/')

# upload a tree to s3
s3shutil.copytree('/home/users/pics/', 's3://bucket/path/archive/')

# copy between two s3 locations
# same or different bucket
s3shutil.copytree('s3://bucket2/files/someth/', 's3://bucket1/backup/old/')

# delete (recursively) entire prefix

Just released! (December 2023), tree_sync operation:

Only copies files that are missing in the destination. Also deletes extra files.

# sync download
s3shutil.tree_sync('s3://bucket/files/docs/', '/home/myuser/docs')

# sync upload
s3shutil.tree_sync('/home/myuser/files/', 's3://bucket/files/docs-v2/')

# sync two bucket locations
s3shutil.tree_sync('s3://bucket/files/docs/', 's3://bucket2/a/b/c')


s3shutil will notice alone if the location is s3 (starts with s3://) or not All operations have a similar string based API of powerfull one liners

Test Matrix

s3shutil is thoroughly unit tested in all the combinations of:

Python Versions:

  • 3.12

  • 3.11

  • 3.10

  • 3.9

  • 3.8

  • 3.7

And boto3 Versions:

  • 1.33

  • 1.30

  • 1.28

  • 1.27

  • 1.26

  • 1.25

  • 1.24

  • 1.23


Just use it! You can send an email as well All emails are (eventually) answered. Also read the code, fork, open a PR, start a discussion.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3shutil-0.37.tar.gz (17.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page