Skip to main content

Tool for procesing AWS S3 access logs

Project description

A simple tool for parsing AWS S3 access logs. Can either be used from Python or via a command line wrapper.

Installation

Use pip

pip install s3-log-parse

Command Line usage

The s3logparse command line script will reformat logs according to the following conventions

  • All fields are tab separated
  • No fields are quoted
  • Null text fields (represented by -) are empty
  • Timestamp is ISO8601 formatted
  • Null numeric fields are converted to 0

The script can accept a single argument, the name of a log file

s3logparse mys3logfile.txt

piping will also work

cat allmys3logs/* | s3logparse

Python API

The parse_log_lines function iterates over raw log lines and converts them to a namedtuple with the following attributes: bucket_owner, bucket, timestamp, remote_ip, requester, request_id, operation, s3_key, request_uri, status_code, error_code, bytes_sent, object_size, total_time, turn_around_time, referrer, user_agent, version_id

Here is a simple example which extracts IP addresses from logs

from s3logparse import s3logparse

with open('mys3logs.txt') as fh:
    for log_entry in parse_log_lines(fh.readlines()):
        print(log_entry.ip_address)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for s3-log-parse, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size s3-log-parse-0.1.1.tar.gz (2.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page