cloudwatch-log-retention-manager 0.1.4

Usage

Install cwlrm through pip and create a config.yml file within your project.

➜ cwlrm --help
usage: cwlrm [-h] [--version] [-u] [-c CONFIGFILE] [-s] [-sc] [-ic] [-v] [-vv]

CloudWatch logs retention manager

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -u, --update          update settings in AWS
  -c CONFIGFILE, --config CONFIGFILE
                        location of config.yaml
  -s, --show_all        show all log groups in filter
  -sc, --show_cost      total cost per groups
  -ic, --show_individual_cost
                        show storage cost for each log group in filter
  -v, --verbose         set loglevel to INFO
  -vv, --very-verbose   set loglevel to DEBUG

Once installed, all that is needed is a config.yml file containing the log group patterns and retention periods to set. The tool supports a Compliance mode, where the live AWS environment will be validated against the described config file.

When you are ready to apply the changes, run cwlrm -u to enforce Compliance as defined.

The format of the config.yml file is as follows:

• A single root object names retentionPatterns

• An array of child objects consisting of the following attributes:

1. name - mandatory - names a section of log groups to be processed

2. logPrefix - optional - the pattern of logs to apply the group to - default: all logs

3. retentionInDays - optional - the log retention period to set - default: Forever

4. override - optional - if set will require the exact retentionInDays values to be compliant, otherwise any log retention is considered compliant - default: false

5. showAlways - optional - when set, will show all log groups that match irrespective of their compliance - default: false

Optionally the logPrefix can be an array of patterns to be collected into the matching group.

Example:

retentionPatterns:
  - name: CodeBuild jobs
    logPrefix: /aws/codebuild/
    retentionInDays: 14
    override: true
    showAlways: true

  - name: AWS Glue Crawlers
    logPrefix: /aws-glue/crawlers
    retentionInDays: 14

  - name: Specific application lambda logs
    logPrefix:
     - /aws/lambda/Application1
     - /aws/lambda/Application2
     - /aws/lambda/Application3
    retentionInDays: 45

  - name: Lambda logs
    logPrefix: /aws/lambda
    retentionInDays: 30

  - name: API Gateway access logs
    logPrefix: /aws/api-gateway/
    retentionInDays: 120

  - name: API Gateway Execution Logs
    logPrefix: API-Gateway-Execution-Logs
    retentionInDays: 120

  - name: All remaining log groups
    showAlways: true
    retentionInDays: 90

IMPORTANT Note: For each run a log group will only be processed by a single control - the first control to process the log group will be the active one, if subsequent patterns are to match that same log group it will be ignored.

Examples

Some example config files.

retentionPatterns:
  - name: AWS Glue Crawlers
    logPrefix: /aws-glue/crawlers
    retentionInDays: 14

  - name: CloudFront Lambda Function logs
    logPrefix: /aws/cloudfront/LambdaEdge
    retentionInDays: 30

retentionPatterns:
  - name: CodeBuild jobs
    logPrefix: /aws/codebuild/
    retentionInDays: 14
    override: true

retentionPatterns:
  - name: Application audit logs
    logPrefix: /application/audit/
    showAlways: true

retentionPatterns:
  - name: API Gateway access logs
    logPrefix: /aws/api-gateway/
    retentionInDays: 30

  - name: Lambda API logs
    logPrefix: /aws/lambda/
    retentionInDays: 30

  - name: API Gateway Execution Logs
    logPrefix: API-Gateway-Execution-Logs
    retentionInDays: 30

  - name: All remaining log groups
    showAlways: true
    retentionInDays: 90

Note

This project has been set up using PyScaffold 4.0.1. For details and usage information on PyScaffold see https://pyscaffold.org/.