Skip to main content

Pulls (filtered) files from S3 and adds them to a tar archive.

Project description

s3tar

Pulls (filtered) files from S3 and adds them to a tar archive.

Creates the command line script star.

$ star --help

Usage: star [OPTIONS] PATH

  Generates a tar archive of S3 files.

  Files are selected by a path made up of 'bucket/prefix' and optionaly by a
  time-based and/or name filter.

  'profile' is the AWS CLI profile to use for accessing S3.  If you use
  chaim or cca then this is the alias name for the account.

  The time based filter relies on the files being named with ISO Formatted
  dates and times embedded in the file names.  i.e.
  'file.2020-03-04T12:32:21.txt' The regular expression used is:

      /.*[._-]{1}([0-9-]{10}T[0-9:]{8}).*/

  The 'start' and 'end' parameters can either be ISO formatted date strings
  or unix timestamps.  If only the date portion of the date/time string is
  given the time defaults to midnight of that day.

  The length parameter is a string of the form '3h', '2d', '1w' for,
  respectively 3 hours, 2 days or 1 week.  Only hours, days or weeks are
  supported.  The 'length' and 'end' parameters are mutually exclusive, give
  one or the other, not both.

  If neither the 'end' nor the 'length' parameter is given, the end time
  defaults to 'now'.

  If the 'start' parameter is not given no time filtering of the files is
  performed, and all files found down the path are copied across to the tar
  archive recursively.

  To use the last modified time stamp of the files rather than their names
  for filtering pass the '-M' flag.

  To use the name filter, pass in a partial string that object names must
  contain.

  The tar archive can be compressed using gzip, bzip2 or lzma. Defaults to
  gzip. Pass a one char string to the `-c` option of "g", "b", "z" or "n".
  "n" is no compression. The output tar archive will be named accordingly:
  ".tar.gz" for gzip, ".tar.bz2" for bzip2, ".tar.xz" for lzma and ".tar"
  for no compression.

  The output filename of the tar archive will be $HOME/<bucket name>.tar
  You can change this with the "-o" option.

  Using the "-q" switch will turn off all messages (except errors) apart
  from the final output of the full path of the tar archive that is created.

  Using the "-v" switch will make the program verbose, showing each file
  that is copied into the tar archive.

  Files in Glacier and Glacier Deep Archive are ignored.

Options:
  -c, --compression TEXT  optional compression ['b', 'g', 'n', 'z'], default
                          'g'

  -e, --end TEXT          optional end time
  -l, --length TEXT       optional time length (i.e. 1d, 3h, 4w)
  -M, --usemodified       use last modified time stamp rather than filename
                          for filtering

  -N, --name TEXT         optional name filter
  -o, --output TEXT       output file name (default: bucket name)
  -p, --profile TEXT      AWS CLI profile to use (chaim alias)
  -q, --quiet             be very quiet, only show the tar file name
  -s, --start TEXT        optional start time
  -v, --verbose           show files that are being copied
  --help                  Show this message and exit.

Install

The script is python3 only (>=python3.6).

Install it under your python3 user directories with:

python3 -m pip install s3tar --user

If this is the first python3 user script you have you will have to adjust your path. The script location will be $HOME/.local/bin on a Linux machine, so add that to you path in your shell init file e.g.

echo "export PATH=$HOME/.local/bin:$PATH" >>~/.bashrc

If your shell is bash.

To check that installed ok:

star --help

Should display the help text.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3tar-1.4.1.tar.gz (5.9 kB view details)

Uploaded Source

Built Distribution

s3tar-1.4.1-py3-none-any.whl (6.6 kB view details)

Uploaded Python 3

File details

Details for the file s3tar-1.4.1.tar.gz.

File metadata

  • Download URL: s3tar-1.4.1.tar.gz
  • Upload date:
  • Size: 5.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.5 CPython/3.8.3 Linux/5.6.16-1-MANJARO

File hashes

Hashes for s3tar-1.4.1.tar.gz
Algorithm Hash digest
SHA256 4307ce10ab5e8b0d7188d1bb6da96a2138d13194d7b86deeed514115ece28da6
MD5 e1ed9f0158fc3e9364a93e72b7bb3d4f
BLAKE2b-256 c8f0c2469c36c6ec6ae908719dc7373c36d49ceaf26d628cb65652ab2779d247

See more details on using hashes here.

File details

Details for the file s3tar-1.4.1-py3-none-any.whl.

File metadata

  • Download URL: s3tar-1.4.1-py3-none-any.whl
  • Upload date:
  • Size: 6.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.5 CPython/3.8.3 Linux/5.6.16-1-MANJARO

File hashes

Hashes for s3tar-1.4.1-py3-none-any.whl
Algorithm Hash digest
SHA256 b0e40f11710cfda60173b2971d8d72e2a472ec83d879d1dae15b3317e36c904b
MD5 9344e54ad81c67fdd65f2dfd12925304
BLAKE2b-256 94830bde3ff5311a663b0795d352e3263719a56b1f744ddcd5f136b66233edf7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page