Skip to main content

Tools for processing logs from AWS Elastic Load Balancer

Project description

ELB Log Tools

Tools for processing logs from AWS Elastic Load Balancer

Installation

$ pip install elb-log-tools

Usage

Specifying the log bucket name

There are two options:

  1. Set the ELB_LOG_BUCKET environment variable, or

  2. Pass the bucket name to elb-logs using the -b or --bucket argument.

    $ elb-logs -b my-log-bucket
    

Print ELB logs to stdout

$ elb-logs | head -n 1
2019-11-06T23:00:47.799875Z production-pymcore-webserver 84.10.79.236:55801 172.16.11.238:80 0.000049 1.922708 0.000054 200 200 0 230454 "GET https://www.pymetrics.com:443/results/downloads/traits/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

Filter ELB logs

Filtering currently only outputs the PATH component of the log file.

# Only output redirects
$ elb-logs | elb-filter --include-statuses 301 302 | head -n 1
/logout/

Aggregate request counts based on pattern matching

elb-patterncounts matches URL paths against a list of regex patterns, and outputs the counts of requests matching the patterns in CSV format.

# patterns.txt has all the URL patterns we want included in the report

$ elb-logs | elb-filter -f 404 | elb-patterncounts --patternfile patterns.txt --all
Path,Hits
/api3/apps/applications/,66762
/api3/games/games/,42192
/api/game/submission/,35418
/api2/games/list/,33487
/api2/accounts/config/,19251
/health/,18891

Examples

Get most common requests for static files that 404:

$ elb-logs | elb-filter -i 404 | grep static | head -n 50 | uniq -c | sort -bgr

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

elb-log-tools-0.3.0.tar.gz (10.0 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page