Skip to main content

s3-restore, a point in time restore tool for Amazon S3

Project description

S3 point in time restore

This is the repository for s3-restore, a point in time restore tool for Amazon S3.

The typical scenario in which you may need this tool is when you have enabled versioning on an S3 bucket and want to restore some or all of the files to a certain point in time, to local file system, same s3 bucket or different s3 bucket.

Doing this with the web interface is time consuming: Amazon S3 web management gui doesn't offer a simple way to do that on a massive scale.

With this tool you can easily restore a repository to a point in time with a simple command like:

  • To local file-system:
     $ s3-restore -b my-bucket -d restored-bucket-local -t "06-17-2016 23:59:50 +2"
    
  • To s3 bucket:-
     $ s3-restore -b my-bucket -B restored-bucket-s3 -t "06-17-2016 23:59:50 +2"
    

Choosing the correct time and date to restore at is simply a matter of getting that information clicking the Versions: Show button from the S3 web gui and navigating through the, now appeared, versions timestamps.

Installing

With pip install:

$ pip3 install s3-restore

or clone the repository and launch:

$ python3 setup.py install

Requirements

  • Python 3
  • AWS credentials available in the environment
    • This can be accomplished in various ways:
      • Environment Variables:
        • AWS_ACCESS_KEY_ID
        • AWS_SECRET_ACCESS_KEY
        • AWS_DEFAULT_REGION
      • Your ~/.aws/ files
        • Configured with aws configure

Usage

s3-restore can do a lot of interesting things. The base one is restoring an entire bucket to a previous state:

Restore to local file-system

  • Restore to local file-system directory restored-bucket-local
     $ s3-restore -b my-bucket -d restored-bucket-local -t "06-17-2016 23:59:50 +2"
    
    • -b gives the source bucket name to be restored from
    • -d gives the local folder to restore to (if it doesn't exist it will be created)
    • -t gives the target date to restore to. Note: The timestamp must include the timezone offset.

Restore to s3 bucket

  • Restore to same bucket:

     $ s3-restore -b my-bucket -B my-bucket -t "06-17-2016 23:59:50 +2"
    
    • -B gives the destination bucket to restore to. Note: Use the same bucket name to restore back to the source bucket.
  • Restore to different bucket:-

     $ s3-restore -b my-bucket -B restored-bucket-s3 -t "06-17-2016 23:59:50 +2"
    
  • Restore to s3 bucket with custom virtual prefix [restored object(src_obj) will have key as new-restored-path/src_obj["Key"]] (Using -P flag)

     $ s3-restore -b my-bucket -B restored-bucket-s3 -P new-restored-path -t "06-17-2016 23:59:50 +2"
    

Other common options for both the cases

  • Another thing it can do is to restore a subfolder (prefix) of a bucket:

     $ s3-restore -b my-bucket -d my-restored-subfolder -p mysubfolder -t "06-17-2016 23:59:50 +2"
    
    • -p gives a prefix to isolate when checking the source bucket (-P is used when deal with the destination bucket/folder)
  • You can also speedup the download if you have bandwidth using more parallel workers (--max-workers flag):

     $ s3-restore -b my-bucket -d my-restored-subfolder -p mysubfolder -t "06-17-2016 23:59:50 +2" --max-workers 100
    
  • If want to restore a well defined time span, you can use a starting (-f) and ending (-t) timestamp (a month in this example):

     $ s3-restore -b my-bucket -d my-restored-subfolder -p mysubfolder -f "05-01-2016 00:00:00 +2" -t "06-01-2016 00:00:00 +2"
    

Command line options

usage: s3-restore [-h] -b BUCKET [-B DEST_BUCKET] [-d DEST]
                      [-P DEST_PREFIX] [-p PREFIX] [-t TIMESTAMP]
                      [-f FROM_TIMESTAMP] [-e] [-v] [--dry-run] [--debug]
                      [--test] [--max-workers MAX_WORKERS]
                      [--sse {AES256,aws:kms}]

optional arguments:
  -h, --help            show this help message and exit
  -b BUCKET, --bucket BUCKET
                        s3 bucket to restore from
  -B DEST_BUCKET, --dest-bucket DEST_BUCKET
                        s3 bucket where recovering to
  -d DEST, --dest DEST  path where recovering to on local
  -p PREFIX, --prefix PREFIX
                        s3 path to restore from
  -P DEST_PREFIX, --dest-prefix DEST_PREFIX
                        s3 path to restore to
  -t TIMESTAMP, --timestamp TIMESTAMP
                        final point in time to restore at
  -f FROM_TIMESTAMP, --from-timestamp FROM_TIMESTAMP
                        starting point in time to restore from
  -e, --enable-glacier  enable recovering from glacier
  -v, --verbose         print verbose informations from s3 objects
  --dry-run             execute query without transferring files
  --debug               enable debug output
  --test                s3 pit restore testing
  --max-workers MAX_WORKERS
                        max number of concurrent download requests
  --sse ALGORITHM
                        specify what SSE algorithm you would like to use for the copy

Testing

s3-restore comes with a testing suite. You can run it with:

Restore to local file-system test cases:

`$ ./s3-restore -b my-bucket -d /tmp/ --test`

Restore to s3 bucket test cases:

`$ ./s3-restore -b my-bucket -B restore-bucket-s3 -P restore-path --test` (make sure you have s3 bucket `restore-bucket-s3`)

Run all the test cases:

`$ ./s3-restore -b my-bucket -B restore-bucket-s3 -d /tmp/ -P restore-path --test`

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3-restore-0.1.2.tar.gz (9.8 kB view details)

Uploaded Source

Built Distribution

s3_restore-0.1.2-py3-none-any.whl (9.2 kB view details)

Uploaded Python 3

File details

Details for the file s3-restore-0.1.2.tar.gz.

File metadata

  • Download URL: s3-restore-0.1.2.tar.gz
  • Upload date:
  • Size: 9.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.8.0

File hashes

Hashes for s3-restore-0.1.2.tar.gz
Algorithm Hash digest
SHA256 5c19ced22339d56d47924320529339e5e375224326d0effbdb6c67e4764100d6
MD5 1341752718ef0eadbbe44581d0216253
BLAKE2b-256 5e21d030c32fc0cddd21bd2582c3c377ab1cbaaa4f50e4fcf2ceb5084371e2b0

See more details on using hashes here.

File details

Details for the file s3_restore-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: s3_restore-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 9.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.8.0

File hashes

Hashes for s3_restore-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 b41c184cbf73ebae6534bd83e0ed171f0e27b14e129e7ac00c04b7e4c3c93902
MD5 e1b421a14559d753ceb913cbc85eb5b4
BLAKE2b-256 71fa2ff803d2a9221c608579d883d614eabc480a4ee63c40749b5a90314960e0

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page