Skip to main content

Tool for procesing AWS S3 access logs

Project description

A simple tool for parsing AWS S3 access logs. Can either be used from Python or via a command line wrapper.

Installation

Use pip

pip install s3-log-parse

Command Line usage

The s3logparse command line script will reformat logs according to the following conventions

  • All fields are tab separated

  • No fields are quoted

  • Null text fields (represented by -) are empty

  • Timestamp is ISO8601 formatted

  • Null numeric fields are converted to 0

The script can accept a single argument, the name of a log file

s3logparse mys3logfile.txt

piping will also work

cat allmys3logs/* | s3logparse

Python API

The parse_log_lines function iterates over raw log lines and converts them to a namedtuple with the following attributes: bucket_owner, bucket, timestamp, remote_ip, requester, request_id, operation, s3_key, request_uri, status_code, error_code, bytes_sent, object_size, total_time, turn_around_time, referrer, user_agent, version_id

Here is a simple example which extracts IP addresses from logs

from s3logparse import s3logparse

with open('mys3logs.txt') as fh:
    for log_entry in parse_log_lines(fh.readlines()):
        print(log_entry.ip_address)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3-log-parse-0.1.1.tar.gz (2.9 kB view details)

Uploaded Source

File details

Details for the file s3-log-parse-0.1.1.tar.gz.

File metadata

File hashes

Hashes for s3-log-parse-0.1.1.tar.gz
Algorithm Hash digest
SHA256 2cef0a2f7638c566b7fd7e33e421d300c2ba25fb89bb8b0a353a30f5147453c3
MD5 9904588a3d84d2212522db2435e5a0de
BLAKE2b-256 4a62e0e3c9320897aa33f81d84068d23f2c1929670eac3d8c8dde7621a3e50ad

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page