Skip to main content

A library to parse S3 log files.

Project description

A python library to parse S3 log files.

Warning

Unit tests currently require actual S3 credentials (and a bucket with logs) and can therefore only be run manually. Mocking the relevant parts of boto.s3 is on the roadmap. Contributions are welcome :)

Purpose

Download S3 logs from a bucket, and parse them.

This application does not store the log objects generated and leaves that to other applications.

Install

Eventually from pypi …

Use

To get logs, simply use the Downloader class:

from itertools import chain
from pprint import pprint
from lss3logs.download import Downloader

MY_ACCESS_KEY_ID = 'XXX'
MY_KEY_SECRET = 'XXX'
MY_S3_BUCKET_NAME = 'XXX'

downloader = Downloader(
    connection=None,
    aws_access_key_id=MY_ACCESS_KEY_ID,
    aws_key_secret=MY_KEY_SECRET,
)

# download 10 logs
logs = downloader.download_files(
    MY_S3_BUCKET_NAME,
    prefix='logs/',
    max_logs=1)

entries = [ log.entries for log in logs ]
entries = list(chain.from_iterable(entries))
[pprint(entry.__dict__) for entry in entries]

Running tests

First you need to specify the test config, which contains the AWS credentials and details of bucket tot test with. python-testconfig is used to manage the test configuration.

Copy test_config.ini.sample to test_config.ini (in the same directory) and set correct values:

export NOSE_TESTCONFIG_AUTOLOAD_INI=`pwd`/test_config.ini

To test with nose:

python setup.py nosetests

or running nosetests directly:

nosetests -s --exe

Directly and with coverage:

nosetests -s --exe --with-coverage --cover-package=lss3logs

(Note: the –exe includes python files whoch are executable, so it’s optional if you don’t have any.)

Code quality

Checking code with pylint:

pylint lss3logs

TODO

Credits

The regular expression for parsing the log lines is copied from a script by “kkowalczyk” located at http://code.google.com/p/kjk/source/browse/trunk/scripts/test_parse_s3_log.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ls-s3-logs, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size ls-s3-logs-0.1.2.tar.gz (12.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page