s3stat

This python module uses the really nice goaccess utility to provide you with an amazing Amazon log file analyser tool that is relatively easy to install, and is extremely easy to extend. GOACCESS version needed: 0.8.5 (it doesn’t work with 0.9+)

Installation

pip install s3stat

This installs s3stat.py in your PYTHONPATH in case you would like to run it from the command line.

Quickstart

Install goaccess

You should install goaccess

Generating an AWS user

First you should create a user that has approriate rights to read your log files, and you should have its AWS access keys ready.

Log in to the aws console
Create a new user and select the option to generate an access key for the user
Save the access key and secure keys, as these will be needed soon

Open the Permissions tab for the user, and attach a new user policy. Select custom policy, and copy the following:

{
  "Statement": [
    {
      "Sid": "Stmt1334764540928",
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketLogging",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:ListBucket",
        "s3:PutBucketAcl",
        "s3:PutBucketLogging",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Sid": "Stmt1334764631669",
      "Action": [
        "cloudfront:GetDistribution",
        "cloudfront:GetDistributionConfig",
        "cloudfront:GetStreamingDistribution",
        "cloudfront:GetStreamingDistributionConfig",
        "cloudfront:ListDistributions",
        "cloudfront:ListStreamingDistributions",
        "cloudfront:UpdateDistribution",
        "cloudfront:UpdateStreamingDistribution"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

Set up logging in your buckets

First you should ask Amazon to generate logs for your buckets and cloudfront distributions.

Run this script

s3stat.py <aws key> <aws secret> <bucket> <log_path>

This will download all the log files for today, and start a goaccess instance in your console.

For further options you might run:

s3stat.py -h

Extending

Actually s3stat was designed to be easy to add to your pythonic workflow, as a result it defines a single class that you can subclass to process the results in json format.:

import s3stat

class MyS3Stat(s3stat.S3Stat):

    def process_results(self, json):
        print json

    def process_error(self, exception, data=None):
        print data
        raise exception

mytask = MyS3Stat(bucket, log_path, for_date, (aws_key, aws_secret))
mytask.run()

Where the aws_* parameters are optional, if missing then they are taken from the environment variables as provided by boto. The process_error method currently is called only when the JSON decoding fails, thus data is the non-decodeable string, while exception is the ValueError raised by Python.

ToDo

provide a command that adds logging to specified buckets and cloudfront distributions

Metadata

Release files for s3stat 2.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for s3stat 2.3.1
File	Size	Uploaded
s3stat-2.3.1.tar.gz	5.7 kB	May 13, 2015	Details

Release files / s3stat-2.3.1.tar.gz

Download URL	s3stat-2.3.1.tar.gz
Size	5.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`80f6e933a022137493c5219ecb134ed79f66e0e75d5cd5e0478b5c73789e5b41`
BLAKE2b-256 checksum How to use checksums	`3c597193b1991c23ed57c96ff7988500fe918bcca243b33f4383b9b0306264c7`
Upload date	May 13, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No