cloudwatch-log-retention-manager 0.1.4

Tooling to manage the Retention Compliance for AWS CloudWatch logs

CloudWatch Logs Retention Manager is a configurable tool that can be used to validate or enforce CloudWatch log retention rules.

Unlike other tools in this space, the cwlrm provides flexibility through the use of configuration, this configuration can apply to a subset of log groups - useful when working in shared environments.

Usage

Install cwlrm through pip and create a config.yml file within your project.

➜ cwlrm --help
usage: cwlrm [-h] [--version] [-u] [-c CONFIGFILE] [-s] [-sc] [-ic] [-v] [-vv]

CloudWatch logs retention manager

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -u, --update          update settings in AWS
  -c CONFIGFILE, --config CONFIGFILE
                        location of config.yaml
  -s, --show_all        show all log groups in filter
  -sc, --show_cost      total cost per groups
  -ic, --show_individual_cost
                        show storage cost for each log group in filter
  -v, --verbose         set loglevel to INFO
  -vv, --very-verbose   set loglevel to DEBUG

Once installed, all that is needed is a config.yml file containing the log group patterns and retention periods to set. The tool supports a Compliance mode, where the live AWS environment will be validated against the described config file.

When you are ready to apply the changes, run cwlrm -u to enforce Compliance as defined.

The format of the config.yml file is as follows:

• A single root object names retentionPatterns

• An array of child objects consisting of the following attributes:

1. name - mandatory - names a section of log groups to be processed

2. logPrefix - optional - the pattern of logs to apply the group to - default: all logs

3. retentionInDays - optional - the log retention period to set - default: Forever

4. override - optional - if set will require the exact retentionInDays values to be compliant, otherwise any log retention is considered compliant - default: false

5. showAlways - optional - when set, will show all log groups that match irrespective of their compliance - default: false

Optionally the logPrefix can be an array of patterns to be collected into the matching group.

Example:

retentionPatterns:
  - name: CodeBuild jobs
    logPrefix: /aws/codebuild/
    retentionInDays: 14
    override: true
    showAlways: true

  - name: AWS Glue Crawlers
    logPrefix: /aws-glue/crawlers
    retentionInDays: 14

  - name: Specific application lambda logs
    logPrefix:
     - /aws/lambda/Application1
     - /aws/lambda/Application2
     - /aws/lambda/Application3
    retentionInDays: 45

  - name: Lambda logs
    logPrefix: /aws/lambda
    retentionInDays: 30

  - name: API Gateway access logs
    logPrefix: /aws/api-gateway/
    retentionInDays: 120

  - name: API Gateway Execution Logs
    logPrefix: API-Gateway-Execution-Logs
    retentionInDays: 120

  - name: All remaining log groups
    showAlways: true
    retentionInDays: 90

IMPORTANT Note: For each run a log group will only be processed by a single control - the first control to process the log group will be the active one, if subsequent patterns are to match that same log group it will be ignored.

Examples

Some example config files.

Simple retention pattern

retentionPatterns:
  - name: AWS Glue Crawlers
    logPrefix: /aws-glue/crawlers
    retentionInDays: 14

  - name: CloudFront Lambda Function logs
    logPrefix: /aws/cloudfront/LambdaEdge
    retentionInDays: 30

This configuration will ensure that all glue-crawler and lambdaedge function logs have a retention set, when applied will set 15 and 30 days respectivly.

Specific compliance retention period

retentionPatterns:
  - name: CodeBuild jobs
    logPrefix: /aws/codebuild/
    retentionInDays: 14
    override: true

This configuration will enforce that all codebuild log groups have specifically a 14 day retention period - any other value will be considered “Non Compliant”

Logs that should be retained Forever

retentionPatterns:
  - name: Application audit logs
    logPrefix: /application/audit/
    showAlways: true

If a log group contains some sort of audit required for long term archive, Compliance can be achieved by specifiying the logGroup pattern and omitting the retentionInDays parameter. In this scenario log groups without retention periods are considered Compliant, and are configured to be shown on each execution.

Show all remaining log groups

retentionPatterns:
  - name: API Gateway access logs
    logPrefix: /aws/api-gateway/
    retentionInDays: 30

  - name: Lambda API logs
    logPrefix: /aws/lambda/
    retentionInDays: 30

  - name: API Gateway Execution Logs
    logPrefix: API-Gateway-Execution-Logs
    retentionInDays: 30

  - name: All remaining log groups
    showAlways: true
    retentionInDays: 90

A normal operation of cwlrm will only process the specified log groups (as selected by the logPrefix) - in some cases you may want to apply a default to the entire account, this can be done through the omission of the logPrefix attribute. Any log group matching the previous conditions will

Note

This project has been set up using PyScaffold 4.0.1. For details and usage information on PyScaffold see https://pyscaffold.org/.