s3streamcat 0.1.5

Streaming based unix cat like functionality for file on S3. Supports gzip, bzip and xz compressed files as well

S3 Stream Cat powers you to instantly check the content of S3 file. Especially it supports streaming and printing data from compressed files.

Compression Format Supported

• Gzip

• XZ

• BZIP

You might find it most useful when you don’t actually want to download the file however want to check the sample data, or to grep for few records matching certain search. Typical usage often looks like this:

s3streamcat s3://bucketname/dir/file_path
s3streamcat s3://bucketname/dir/file_path | more
s3streamcat s3://bucketname/dir/file_path | grep something

To try different encoding in case utf-8 is unable to decode your stream::

s3streamcat -e latin-1 s3://bucketname/dir/file_path.gz

To tune number of bytes to be fetched at a time::

s3streamcat -b 1024 s3://bucketname/dir/file_path.gz

Configurations

If you have aws client installed on your system s3streamcat will work out of the box If not you will have to have file at location HOME/.aws/credentials with aws cred:

[default]
aws_access_key_id=<put your aws access key>
aws_secret_access_key=<put your aws secret key>

Dependancies

• Its written for Python3 which is the present and future of the language

• libssl-dev libffi-dev

• python3-dev

Ubuntu users can install dependancy packages with:

• sudo apt-get install -y libssl-dev libffi-dev

• sudo apt-get install python3-dev