Skip to main content

A structured JSON logger that seamlessly interoperates with AWS Cloudwatch and other AWS services (e.g. Lambda)

Project description

Overview

This package provides structured JSON logging faculties designed to seamlessly interoperate with log aggregators like AWS Cloudwatch, ELK, and others. These faculties are provided via extension of the python's existing logging library and can be used as a drop in replacement.

Usage:

If you want to generate simple structured logs, use this package just like the Python logging library.

If you want inject contextual information into your structured logs, add a LogAppender. This is great option if you want to run your application in somewhere like AWS Lambda, where you would want your logs to contain aws_request_id and other function specific metadata. See Use Case: AWS Lambda for more.

If you want to ensure that unwanted information is removed from your logs, use a LogFilter. Filters are a great way to trim the size of your log lines to cut down on logging costs. Filters can also be useful in specific log stacks - namely the ELK stack - that need to inject specific keys like the type key to index logs correctly.

Filters and Appenders can be mixed and matched through your logger's addFilter and removeFilter methods. Appenders and Filters are both dynamically configurable via the logger's update_appender and update_filter methods.

Basic use: JSON logging

import logging
from cloudwatch_logging import CloudwatchLogging

# create a structured JSON logger that works well with Cloudwatch and others; optionally with runtime specific appenders and filters to remove unwanted fields
logger = CloudwatchLogging.create_logger(__name__, appender=None, filter=None)
logger.setLevel(logging.INFO)
logger.warning("Structured logging with custom fields", extra={"custom_field": "custom_value"})

Use case; AWS Lambda

import logging
from cloudwatch_logging import CloudwatchLogging

# Setup logging
logger = CloudwatchLogging.create_logger("your_lambda_function")
logger.propagate = False  # disable Lambda runtime default logger from double logging lines sent to this logger
logger.setLevel(logging.INFO)

def your_lambda_handler(event, context):
    lamdbda_appender = CloudwatchLogging.LogAppender(context)
    logger.addFilter(lamdbda_appender)
    logger.info("This line will be logged with info from the Lambda context object!", extra={"tapped_in": True})
    logger.removeFilter(lamdbda_appender)  # ensure you do this so you don't add duplicate appenders on the same logger

Use case; Cloudwatch Cost Optimization

import logging
from cloudwatch_logging import CloudwatchLogging, Filters

logger = CloudwatchLogging.create_logger("your_lambda_function")
logger.setLevel(logging.INFO)

# add a LogTrimmer on the logger and save some money on CW storage costs
logger.addFilter(CloudwatchLogging.LogFilter(Filters.COST_EFFECTIVE))
logger.warning("Structured logging on the cheap!", extra={"field": "value"})

Development

Setup

This is a pure Python library; it can easily be installed using pip install cloudwatch_logging. Use the -e arg if you plan to develop locally.

Test

Run python3 test_cloudwatch_logging.py. Inspect the output to see what is logged. Someone should write proper tests but I wrote this on vacation and couldn't be bothered.

To-Do

  • Check error and exception trace handling - how can we make these better?
  • Add examples for other AWS runtimes (Fargate?)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cloudwatch-logging-1.0.0.tar.gz (4.4 kB view details)

Uploaded Source

Built Distribution

cloudwatch_logging-1.0.0-py3-none-any.whl (9.9 kB view details)

Uploaded Python 3

File details

Details for the file cloudwatch-logging-1.0.0.tar.gz.

File metadata

  • Download URL: cloudwatch-logging-1.0.0.tar.gz
  • Upload date:
  • Size: 4.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.7.8

File hashes

Hashes for cloudwatch-logging-1.0.0.tar.gz
Algorithm Hash digest
SHA256 10cc937baf258959cbfe1f4995e356b7eeab3a567330ff0dbffbe26890305d8d
MD5 14ceaf84b1bac63ae437b199e3124888
BLAKE2b-256 7c6af95a5d1c83793dfff331b7cf7fac34aebb80e6b06495eb2e238727f06d8b

See more details on using hashes here.

File details

Details for the file cloudwatch_logging-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: cloudwatch_logging-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 9.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.7.8

File hashes

Hashes for cloudwatch_logging-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 31403fd97a9aedeee64e2a488a2e2045c3ca531f06dd540af43c3cf2ad4dea3f
MD5 c49e0227f7b61dda70b81a7091c54321
BLAKE2b-256 d9da69ab907791f1143a50902c12ad0ec7c30c0521d1ba27b1466446777b3746

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page