LOGGING EXTRAS

A collection of various python logging extensions.

Documentation

https://logging-extras.readthedocs.io/en/latest/

Changelog

See CHANGELOG.rst for the release history.

Installation

Install logging-extras using pip

pip install logging-extras

Alternatively, download the latest binary or source package from github

Install wheel package with pip:

pip install logging_extras-{tags}.whl

Install source package as editable:

tar -xf logging-extras-{tags}.tar.gz
cd logging-extras-{tags}
pip install -e .

Please refer to documentation pages for available modules.

Module Index

config.YAMLConfig

YAMLConfig class can be used for loading YAML files with custom tags. This class adds a custom envvar tag to native YAML parser which is used to evaluate environment variables. Supports one or more environment variables in the form of ${VARNAME} or ${VARNAME:DEFAULT} within a string. If no default value is specified, empty string is used. Default values can only be treated as plain strings. YAMLConfig can also expand ~ or ~username just like shells do, either directly hardcoded in YAML file or passed through environment variables.

Example configuration:

File: logging.yaml

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: ${LOGGING_ROOT:.}/${LOG_FILENAME}
    formatter: simple
loggers:
  test_logger:
    level: DEBUG
    handlers:
      - file_handler
    propagate: no
root:
  level: NOTSET
  handlers:
    - console

Note: Ignore the backslashes as markdown must display those escape characters.

Example Usage

File: test_logger.py

import logging
from logging_.config import YAMLConfig

with open("logging.yaml", "r") as config_file:
    YAMLConfig(config_file.read(), silent=True)

# alternatively, you can use
# YAMLConfig.from_file("logging.yaml", silent=True)

logger = logging.getLogger("test_logger")

logger.debug("This is a debug log")
logger.info("This is an info log")
logger.warning("This is an warning log")
logger.error("This is an error log")
logger.critical("This is a critical log")

Note: An (optional) explicit silent=True flag must be set to suppress any file or parsing related exceptions to be thrown.

handlers.QueueListenerHandler

Note: On Python 3.12+, the standard library's logging.config.dictConfig supports queue logging natively (see Configuring QueueHandler and QueueListener), so users on 3.12+ can — and generally should — use the stdlib directly.

logging-extras's QueueListenerHandler is primarily a backport for Python 3.8 – 3.11 that provides the same end-to-end dictConfig-based queue setup on those older runtimes.

A simple queue-logging handler utilizing QueueListener for configured handlers (intentionally subclasses logging.Handler, not logging.handlers.QueueHandler, so Python 3.12+ dictConfig does not special-case construction). This is helpful for detaching the logger handlers from the main threads, which reduces the risk of getting blocked, for example, when using slower handlers such as smtp, file, or socket handlers.

Example configuration:

File: logging.yaml

version: 1
objects:
  queue:
    class: queue.Queue
    maxsize: 1000
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: test_logger.log
    formatter: simple
  queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.console
      - cfg://handlers.file_handler
    queue: cfg://objects.queue
loggers:
  test_logger:
    level: DEBUG
    handlers:
      - queue_handler
    propagate: no
root:
  level: NOTSET
  handlers:
    - console

Note: A queue object must be passed since the handler does not set a default queue implementation. Set maxsize: -1 to make the queue unlimited.

Example Usage

File: test_logger.py

import logging.config
import yaml

with open("logging.yaml", "r") as config_file:
    logging_config = yaml.safe_load(config_file.read())
    logging.config.dictConfig(logging_config)

logger = logging.getLogger("test_logger")

logger.debug("This is a debug log")
logger.info("This is an info log")
logger.warning("This is an warning log")
logger.error("This is an error log")
logger.critical("This is a critical log")

Development

Additional development and documentation dependencies can be installed using extras. It is recommended to use a virtualenv.

Use Pre-Commit Hooks

Install pre-commit hooks and dependencies:

pip install pre-commit
pre-commit install
pre-commit autoupdate
pre-commit run --all-files

Run Tests

Run tests from the source with Pytest:

pip install -e .[dev]
pytest -s

Generate Documentation

Generate documentation from the source with Sphinx:

pip install -e .[doc]
cd docs
mkdir -p _static _templates
make html
python -m http.server --directory build/html

No requirements.txt File

This is a python library package that is compatible with a wide range of Python versions. It does not make much sense to pin dependency versions in a traditional requirements.txt file. Instead, this project utilizes modern python packaging paradigms with pyproject.toml and setup.cfg files. However, sometimes some IDEs (i.e. PyCharm) cannot resolve dependencies without a requirements.txt file. To generate a requirements.txt file, simply run the following command within your venv:

pip freeze > requirements.txt

Create Distribution Packages

To create a source and wheel distribution, run:

git clone git@github.com:zobayer1/logging-extras.git
python -m pip install wheel
python setup.py clean sdist bdist_wheel

Note: This project uses setuptools-scm to generate build versions from git tags. Build system will raise errors if you are trying to build packages outside a git repo.