modern-pylogging 0.2.0

Helper library for configuring python logging

Modern PyLogging

A library to simplify logging configuration.

Inspired by litestar's logging configuration.

Overview

ModernPyLogging provides a wrapper around Python’s standard logging library,
offering structured logging with JSON output, sensible defaults, and simplified configuration.

It is designed primarily for async applications and allows you to bind context so that every log entry contains it.

It implements the recommended approach to setting up loggers.

There are notable differences in the logging module between Python < 3.12 and >= 3.12
(including a bug in 3.12.4 regarding the queue parameter in QueueHandler).

Due to these variations, I decided to extract my logging configuration into a reusable library for multiple projects.

Features

• 🚀 Simple API – Get started with just a few lines of code
• 🧱 Built on the standard library – Uses Python’s built-in logging module
• 🔍 Structured logging – JSON output for easy parsing and analysis
• 🎨 Console output – Human-readable logs during development
• 🔧 Flexible configuration – Via code or configuration files
• 🔄 Tool integration – Can use orjson and picologging if installed
• 📊 Context management – Easily add context to your logs

Installation

pip install modern-pylogging

Quick Start

import modern_pylogging

config = modern_pylogging.LoggingConfig()
get_logger = config.configure()  # sets up logging and returns a logger getter (like logging.getLogger)

# Default logging level is INFO

# Get a logger for your module
logger = get_logger(__name__)

# Start logging!
logger.info("Application started")
logger.debug("Debug information", extra={"user_id": 12345})
logger.warning("Something might be wrong")
logger.error("An error occurred", exc_info=True)

Default Configuration (Short Version)

• All logs are sent to sys.stdout (useful for Kubernetes or containers).
• All filters, handlers, etc., are attached to the root logger.
• propagate is not disabled – all logs from your app and external libraries are captured by the root logger.
• Default logging level is INFO (can be changed via the LOGGING_LEVEL env variable or manually).
• Logs are output in JSON format (Kubernetes-friendly).
• The environment (env, system, inst) is included in every log record, sourced from environment variables.
• QueueHandler is used to avoid blocking the main thread. It stores log records in a queue, and a corresponding QueueListener processes them separately (helpful for async event loops).
• If orjson is installed, it's used automatically for JSON dumping.
• If picologging is installed, it's used as a drop-in replacement for the standard logging module.
• Both orjson and picologging usage can be configured manually.

Default Configuration (Extended Version)

         MAIN THREAD             │      ANOTHER THREAD   
                                │                       
    APP─────┐                   │                       
           │                    │                       
           │                    │                       
     ┌─────▼──────────────────┐ │  ┌──────────────┐     
     │   Root logger          │ │  │QueueListener │     
     ├────────────────────────┤ │  └▲──────┬──────┘     
     │ Handler: QueueHandler  │ │   │      │            
     │                        │ │   │      │            
     └────┬───────────────────┘ │   │   ┌──▼─────────────┐       
          │                     │   │   │                │       
          │   ┌──────┐          │   │   │     STDOUT     │       
          └──►│QUEUE ├──────────┼───┘   └────────────────┘                
              └──────┘          │                       
                                │                       

Differences Between Python ≤ 3.11 and ≥ 3.12

• Python ≤ 3.11: QueueHandler must have its own Formatter and sends pre-formatted data (usually str) to the queue. QueueListener simply passes them to its handlers (e.g., stdout). So the main thread performs all formatting and JSON serialization.

• Python ≥ 3.12: QueueHandler places LogRecord objects into the queue and must not have a Formatter. QueueListener receives raw LogRecords and formats them in its own thread.

⚠️ Because of this, logging configs must differ depending on your Python version.

Configuration

Basic Configuration

The simplest way to configure logging:

import modern_pylogging

# Configure with sensible defaults
config = modern_pylogging.LoggingConfig()
get_logger = config.configure()

Parameters of LoggingConfig:

logging_module (Literal['logging', 'picologging']):

Specifies the backend logging module to use. 'logging' is the standard library, while 'picologging' is a faster, drop-in replacement. 'picologging' will be used by default if installed.

json_dumps_module (Literal['json', 'orjson']):

Module used to serialize log records to JSON format. 'orjson' provides better performance and type handling. 'orjson' will be used by default if installed.

version (Literal[1]):

Schema version for the logging configuration. The only valid value at present is 1

disable_existing_loggers (bool):

Whether to disable loggers that were previously configured before applying this config.

filters (dict[str, dict[str, Any]] | None):

A dict in which each key is a filter id and each value is a dict describing how to configure the corresponding Filter instance. https://docs.python.org/3/library/logging.html#filter-objects.

formatters (dict[str, dict[str, Any]] | None):

A dict in which each key is a formatter and each value is a dict describing how to configure the corresponding Formatter instance. A 'standard' and 'json_fmt' formatter is provided. standard - write everything as strings, good for development. json_fmt - write output as json, good for production. https://docs.python.org/3/library/logging.html#formatter-objects

handlers (dict[str, dict[str, Any]] | None):

A dict in which each key is a handler id and each value is a dict describing how to configure the corresponding Handler_ instance. Two handlers are provided, 'console' and 'queue_handler'. https://docs.python.org/3/library/logging.html#handler-objects

loggers (dict[str, dict[str, Any]] | None):

A dict in which each key is a logger name and each value is a dict describing how to configure the corresponding Logger_ instance. https://docs.python.org/3/library/logging.html#logger-objects

root (dict[str, dict[str, Any] | list[Any] | str] | None):

This will be the configuration for the root logger. Processing of the configuration will be as for any logger, except that the propagate setting will not be applicable.

override_formatters (dict[HandlerName, FormatterName] | None):

Sometimes you want just use default settings but change formatters (e.g. to write to console as text instead of json) You could provide a dict like {'console': 'standard', 'queue_handler': 'standard'}

level (int | str | None):

Optional default logging level to be applied globally (e.g., 'DEBUG', 10).

capture_extra_fields (bool):

Whether to capture additional fields from log records' extra parameter into the structured output (especially relevant for JSON loggers). This does not work for 'picologging'. It's where you use like this - logger.info('abc', extra={'key': 'value'})

Context Management

modern_pylogging.update_log_extra allows you to bind context to the current coroutine.

This context propagates to all child coroutines created with await, TaskGroup, create_task, or gather.

Example:

If the name of parameter has a dot in it - it would be a nested dictionary in json output.

import modern_pylogging

modern_pylogging.update_log_extra({
    'request.id': 'super ID',
    'request.method': 'GET',
})

logger.info('Done processing')
# Extra data is nested in JSON output:
# {'request': {'id': 'super ID', 'method': 'GET'}, ...}

All coroutines in the same asyncio Task share context (if you’re using await). So nested coroutines can modify the shared context.

import modern_pylogging

async def func_a():
    modern_pylogging.update_log_extra({'a': 'AAA'})
    await func_to_call()

async def func_to_call():
    modern_pylogging.update_log_extra({'b': 'BBB'})

Child tasks, however, run with a copy of the context and cannot change the parent’s context. (Applies when using asyncio.gather, create_task, or TaskGroup.)

from asyncio import TaskGroup
import modern_pylogging

async def func_a():
    modern_pylogging.update_log_extra({'a': 'AAA'})
    async with TaskGroup() as tg:
        tg.create_task(func_to_call())

async def func_to_call():
    modern_pylogging.update_log_extra({'b': 'BBB'})

Logging Context Example

import asyncio
import modern_pylogging
import logging

modern_pylogging.LoggingConfig(logging_module='logging').configure()
logger = logging.getLogger('example')

async def main():
    logger.info('log from main')
    modern_pylogging.update_log_extra({'user_id': 42})
    logger.info('log from main [updated]')
    await another_async_func()
    logger.info('log after another_async_function')

async def another_async_func():
    modern_pylogging.update_log_extra({'service': 'k'})
    logger.info('log from another_async_func')
    await asyncio.gather(task1(), task2())

async def task1():
    logger.info('log from task1')

async def task2():
    logger.info('log from task2')

Expected Output:

{"message":"log from main"}
{"message":"log from main [updated]", "user_id":42}
{"message":"log from another_async_func", "user_id":42, "service":"k"}
{"message":"log from task1", "user_id":42, "service":"k"}
{"message":"log from task2", "user_id":42, "service":"k"}
{"message":"log after another_async_function", "user_id":42, "service":"k"}

Overriding Formatters

During development, you may prefer string output instead of JSON in the console. Instead of redefining the entire logger config, use the override_formatters parameter.

queue_handler sends data to console handler and depending on python version you're using it's even queue_handler's Formatter or console's Formatter.

import logging
import os
import modern_pylogging

if os.environ.get('DEV_MODE'):
    override_formatters = {'queue_handler': 'standard', 'console': 'standard'}
else:
    override_formatters = {}

modern_pylogging.LoggingConfig(logging_module='logging', override_formatters=override_formatters).configure()

logger = logging.getLogger('example')
logger.info("Application started")
# Output (in dev mode):
>> 2025-05-09 20:24:08,888 | INFO | __main__:doc_file:13 | Application started |

Configuring Existing Loggers

import modern_pylogging

config = modern_pylogging.LoggingConfig(
    logging_module='logging',
    level='DEBUG',
    loggers={
        'httpx': {'handlers': [], 'level': 'INFO', 'propagate': True},
        'peewee': {'handlers': [], 'level': 'INFO', 'propagate': True},
        'aiokafka': {'handlers': [], 'level': 'INFO', 'propagate': True},
    }
)
config.configure()
# All logs use DEBUG level, except specified loggers which remain at INFO.