merges3logs 0.1

Download cloudfront logs from S3 and merge them into a single log file for the day.

merges3logs

Download cloudfront logs from S3 and merge them into a single log file for the day.

Installation

pip install merges3logs

Usage

Set up the config file as documented below.

Run:

merges3logs path/to/config.ini

This will pick up the logs from yesterday (UTC). To run for a different date, run:

merges3logs path/to/config.ini --date YYYY-MM-DD

When to Run

Some log lines for a given date will show up in the log files dated the following day. This is because the last bit of one day doesn't get flushed until the following day, and the file written will have that days date. These dates are all in UTC. Depending on your logging configuration, it could take hours for all the last logs for a day to get flushed to S3.

So, you probably want to run this program at least a few hours after midnight, UTC.

Any logs you download for the following day are cached and not re-downloaded the next day.

Configuration

The bulk of the configuration is done via a ".ini"-style configuration file. Here is an example:

[AWS]
AccessKey = XXX
SecretKey = XXX

[S3]
BucketName = mylogsbucket
Prefix = path/to/logfileprefix_log-%%Y-%%m-%%d-
MaxWorkers = 10

[Local]
CacheDir = /path/to/mylogsbucketcache
DestDir = /path/to/merged-logs
DestFilename = webworkers-cloudfront-%%Y-%%m-%%d.gz

Details:

• AWS specifies your access and secret keys.
• S3.BucketName is the name of your bucket.
• S3.Prefix is the "prefix" of your log file names for a certain date. An S3 prefix is everything after the bucket name up to and including the date, with the date encoded using "strftime()" format, however the "%"s need to be doubled (because INI format otherwise interprets them).
• S3.MaxWorkers is the number of download jobs that will run to get logs. Depending on your logging configuration in Cloudfront and how widely your services are accessed, this can be tens or hundreds of thousands of log files a day. So running downloads in parallel can really speed it up.
• Local.CacheDir is the path to a directory to store the downloaded log files. This directory will need to have a cleanup job set up to prevent it from blowing up.
• Local.DestDir is the directory that the merged log files will be written to.
• Local.DestFilename is the name of the file that will be written in the DestDir with "strftime()" format to specify the date.

Cleanup

merges3logs will download the log files into a cache directory, and then work from the files there. You will need a cleanup job to prevent that from growing, for example:

find /path/to/cachedir -type f -mtime +3 -exec rm {} +

You will also need to clean up the destination log directory, though it does grow much more slowly (logs are compressed and only one file per day). Something like using logrotate or:

find /path/to/merged-logs -type f -mtime +60 -exec rm {} +

Author

Written by Sean Reifschneider, Oct 2023.

License

CC0 1.0 Universal, see LICENSE file for more information.