ucam-observe 0.3.0

Python library to aid consistent configuration of logging, metrics (future) and tracing (further in future). Packaging and wiring existing open tooling to work effortlessly on UIS DevOps managed cloud infrastructure.

Observability Python Library

Currently a POC in the Wilson team. Python library to aid consistent configuration of logging, metrics (future) and tracing (further in future). Packaging and wiring existing open tooling to work effortlessly on UIS DevOps managed cloud infrastructure.

ucam_observe integrates with gunicorn, django and plain Python projects. It expects that gunicorn is used to serve both Django and plain-python web app projects.

• Install this module
• Usage
  • Logging
  • Metrics and Tracing
• Environment Configuration
  • Log Level
  • Console Logging
  • Example Docker Compose Configuration
• Gunicorn setup
  • Adapt Gunicorn configuration
• Django project setup
  • Django Settings
  • Console Logging and DEBUG
  • External Settings and Environment Variables
• Developing ucam_observe
  • Developer quickstart
  • Optional extras

Install this module

pip install ucam-observe          # For any python project
pip install ucam-observe[django]  # for django projects

Once installed:

• Web apps running in gunicorn must configure gunicorn
• Django projects must configure django

Usage

Logging

Usage is similar to using structlog directly with the function get_structlog_logger returning an object compatible with that returned by structlog's get_logger function. No further configuration is needed.

logger = get_structlog_logger(__name__)

logger.info("some_event")

logger.info("some_other_event", foo=bar)

Metrics and Tracing

raise NotImplemented

Environment Configuration

Log Level

Set the LOG_LEVEL environment variable to control the logging level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL). This setting adjusts the verbosity of the log outputs:

export LOG_LEVEL=DEBUG

Console Logging

Set the CONSOLE_LOGGING environment variable to control whether logs should be output in a console-friendly or JSON format. JSON is used in production.

If it's not set, the default behaviour auto-selects (human-readable) console logging when running in an interactive console, and JSON when it's not. As a result, you shouldn't need to set CONSOLE_LOGGING, other than in specific situations, like when testing different outputs.

Set it to True to force console-friendly formatting, or False to force JSON output:

export CONSOLE_LOGGING=True

Example Docker Compose Configuration

When using Docker Compose for local development, you can set the environment variables in your docker-compose.yml file:

services:
  your_service:
    build: .
    environment:
      LOG_LEVEL: "DEBUG"
      # Or, to allow the calling environment to override LOG_LEVEL:
      # LOG_LEVEL: "${LOG_LEVEL:-DEBUG}"

Gunicorn setup

Adapt Gunicorn configuration

In the root of your project, create/amend a gunicorn.conf.py. Add the following code to the file.

logger_class = "ucam_observe.gunicorn.UcamObserveLogger"

You don't have to set any other logging configuration options.

If you want to adjust the logging config, you can extend the default config like this:

import ucam_observe.gunicorn

logger_class = "ucam_observe.gunicorn.UcamObserveLogger"
logconfig_dict = {
    **(default_config := ucam_observe.gunicorn.get_gunicorn_dict_config()),
    "loggers": {
        **default_config["loggers"],
        "custom": {
            "level": "ERROR",
        },
    },
}

Django project setup

Django Settings

1. Include "ucam_observe" and "django_structlog" in INSTALLED_APPS
2. Include "django_structlog.middlewares.RequestMiddleware" in MIDDLEWARE
3. Set LOGGING_CONFIG to None (this disables django's builtin logging initialisation)
4. (Optional) modify LOGGING to extend ucam_observe's default logging config

LOGGING_CONFIG = None  # disable Django logging configuration in favour of ucam-observe

INSTALLED_APPS = [
    ...,
    "ucam_observe",
    "django_structlog",
]

MIDDLEWARE = [
    ...,
    "django_structlog.middlewares.RequestMiddleware",
]

Or if you want to modify the logging config:

from ucam_observe.django import get_django_dict_config

# use the `LOGGING` setting as normal, but extend the default config:
LOGGING = get_django_dict_config()
LOGGING["loggers"]["foo.bar"] = {"level": "ERROR"}

This disables Django default logging configuration behaviour and defers all logging configuration to ucam-observe. This ensures logging configuration is only configured once and the surplus default Django loggers are not added.

Console Logging and DEBUG

The Django convention is to log to the console when DEBUG=True. ucam_observe always logs to stdout/stderr and detects whether its running in an interactive console to switch between human-readable or JSON structured log output. See the Console logging section for details.

External Settings and Environment Variables

ucam_observe does not support environment variables from externalsettings, for example EXTERNAL_SETTING_LOG_LEVEL will not configure the logging level. Environment variables must be as documented above.

---

Developing ucam_observe

Everything below is for developers working on ucam_observe itself, people using the library can ignore this.

Developer quickstart

Firstly, install docker-compose.

Install poethepoet

pip install poethepoet

Then, most tasks can be performed via the poe command.

E.g.

# Build the containers
$ poe build

Run the follow command to see available commands:

$ poe

Optional extras

This library includes optional extras, e.g. ucam-observe[django]. Some tests will require these optional dependencies to pass. The following command will install all optional dependencies.

$ poetry install --all-extras --with django-dev

Some tests require the absence of dependencies and these are excluded by default. See the tox.ini file for how these tests are run.