Skip to main content

A tool to get all files from an S3 bucket

Project description

Sucket

A tool to download all objects from an S3 bucket. Supports filtering keys by a prefix.

Help

-> % sucket --help
Usage: sucket [OPTIONS] BUCKET_NAME [PREFIX]

  Download all files from a S3 bucket

  Everything from the bucket BUCKET_NAME is downloaded, with an optional key
  filter, specified with PREFIX

  By default the "folder" mode is used, which will keep the bucket "folder
  structure" when downloading.

  The "flat" mode will download all objects into the current folder by
  adding the folder structure into the key name.

  The "keys-only" will completely disregard the folders and put all files in
  the current folder.

Options:
  -m, --mode [folder|flat|keys-only]
                                  The structure to download the objects in.
  -y, --yes                       Don't prompt for continuing
  -q, --quiet                     Don't print out any info, assumes --yes
  -s, --semaphores INTEGER        Max number of asynchronous requests to make.
                                  Default: 1000
  --help                          Show this message and exit.

Examples

folder flat keys-only

Notes

This has only been tested on a fairly limited set of data, but has worked well for ~3500 small files. Need to tweak and experiment with larger files.

Special thanks

This is heavily based on a script that @steinnes wrote some time ago and has been very useful to me for a long time, the name was entirely his idea.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sucket, version 0.3.4
Filename, size File type Python version Upload date Hashes
Filename, size sucket-0.3.4-py3-none-any.whl (5.1 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size sucket-0.3.4.tar.gz (4.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page