payla-utils 0.2.6

payla_utils python package

payla_utils python package

Features

Structlog config

Example, structlog configuration, django

in django settings.py

from payla_utils.logging import LoggingConfigurator

LoggingConfigurator(
    'testapp',
    log_level='INFO',
    own_apps=settings.OWN_APPS,
    setup_logging_dict=True,
).configure_structlog(formatter='plain_console')

Example, structlog configuration, passing extra loggers names

in django settings.py

from payla_utils.logging import LoggingConfigurator

LoggingConfigurator(
    'testapp',
    log_level='INFO',
    own_apps=settings.OWN_APPS,
    setup_logging_dict=True,
).configure_structlog(formatter='plain_console', extra_loggers_name=['mylogger1', 'mylogger2'])

If you want to use structlog in django celery

in celery.py

from django.conf import settings
from payla_utils.logging import LoggingConfigurator

@signals.setup_logging.connect
def receiver_setup_logging(
    loglevel, logfile, format, colorize, **kwargs
):  # pragma: no cover

    LoggingConfigurator(
        'testapp',
        log_level='INFO',
        own_apps=settings.OWN_APPS,
        setup_logging_dict=True,
    ).configure_structlog(formatter='plain_console')

If you want to use structlog with Sentry

You will have to set PaylaLoggingIntegration in sentry sdk setup

sentry_sdk.init(
    # take sentry DSN from environment or add a default value here:
    dsn=env.str('SENTRY_DSN'),
    integrations=[DjangoIntegration(), CeleryIntegration(), PaylaLoggingIntegration()],
    auto_session_tracking=False,
    traces_sample_rate=0.01,
    send_default_pii=True,
    attach_stacktrace=True,
    request_bodies='medium',
    release=PROJECT_VERSION,
    environment=ENVIRONMENT,
)

If you want to use a structlog, not Django based project

from payla_utils.logging import LoggingConfigurator

LoggingConfigurator(
    'testapp',
    log_level='INFO',
    own_apps=[],
    setup_logging_dict=True,
).configure_structlog(formatter='plain_console')

How to use generic structured logger:

logger = structlog.get_logger(__name__)
logger.warning("Here is your message", key_1="value_1", key_2="value_2", key_n="value_n")

Using request middleware to inject request_id and/or trace_id:

This middleware will inject reqest_id and/or trace_id to all logs producer during a request/response cycle.

MIDDLEWARE += ['payla_utils.logging.middlewares.RequestMiddleware']

• Pass your custom request id header via the PAYLA_UTILS settings: REQUEST_ID_HEADER, defaults to X-Request-ID
• Enable tracing (Only supports opentelemetry) via configure_structlog(tracing_enabled=True)

Example of logging with structlog:

Automatic injection of:

• user_id
• IP
• request_id
• trace_id when available

This a console view, in prod it will be json (using python json logging to have standard logging and structlog logging as close as possible)

2022-12-06T12:08:49.296789Z [info     ] This is an info log message    extra_data=test_data_info ip=127.0.0.1 name=payla.urls request_id=56bd5250-aa6e-49c6-b5bc-14c1ec5a9e1f user_id=13
2022-12-06T12:08:49.297239Z [info     ] This is an info log message and should not be an error or errors extra_data=test_data_info2 ip=127.0.0.1 name=payla.urls request_id=56bd5250-aa6e-49c6-b5bc-14c1ec5a9e1f user_id=13
2022-12-06T12:08:49.297473Z [error    ] This is an error log message   extra_data=test_data_error ip=127.0.0.1 name=payla.urls request_id=56bd5250-aa6e-49c6-b5bc-14c1ec5a9e1f user_id=13
2022-12-06T12:08:49.297823Z [critical ] This is a critical log message extra_data=test_data_critical ip=127.0.0.1 name=payla.urls request_id=56bd5250-aa6e-49c6-b5bc-14c1ec5a9e1f user_id=13

How that log is produced?

def logging_test_view(request):
    logger.debug('This is a debug log message', extra_data='test_data_debug')
    logger.info('This is an info log message', extra_data='test_data_info')
    # Let's put the word error into an info message
    logger.info('This is an info log message and should not be an error or errors', extra_data='test_data_info2')
    logger.error('This is an error log message', extra_data='test_data_error')
    logger.critical('This is a critical log message', extra_data='test_data_critical')
    return HttpResponse('Log submitted')

instead of logger = logging.getLogger(__name__) it is logger = structlog.get_logger(__name__)

Behind the scenes? RequestMiddleware is setting structlog contextvars:

def __call__(self, request: HttpRequest) -> HttpResponse:
    ...
    structlog.contextvars.bind_contextvars(ip=ip)
    structlog.contextvars.bind_contextvars(user_id=user_id)
    response = self.get_response(request)

    ...
    structlog.contextvars.clear_contextvars()
    return response

See configuration section

Why structured logger

• By default, the logging frameworks outputs the traces in plain text and tools like EFK stack or Grafana Loki can’t fully process these traces.

• Therefore, if we “structure” or send the traces in JSON format directly, all the tools can benefit of.

• As a developer, it would be nice to be able to filter all logs by a certain customer or transaction.

• The goal of structured logging is to solve these sorts of problems and allow additional analytics.

• When you log something, remember that the actual consumer is the machine Grafana Loki (EFK stack), not only humans.

• Our generic logger comes with some default context structure, but as you can see, you can introduce new keys.

• We use structlog as wraper on standard logging library, you can check for more details structlog.

Access decorator

To prohibit access to only internal IPs for a specific view it's possible to use the only_internal_access decorator.

SERVER_IP is required to be set on payla_utils settings.

See configuration section

Example usage

@only_internal_access
def test_view(request):
    return HttpResponse('OK')

Or inline

path('test/', only_internal_access(test_view), name="test-view"),

Management command to init environment

This management command will init environment based on the current env (local.dev, dev, stage, playground and prod)

• load fixtures on the first run (when the DB is empty)
• setup custom theme for admin_interface
• create user when not in prod if LOCAL_DJANGO_ADMIN_PASSWORD is set

APP_NAME and ENVIRONMENT settings are required. See configuration section

Configuration and settings

Settings for Payla Utils are all namespaced in the PAYLA_UTILS setting. For example your project's settings.py file might look like this:

PAYLA_UTILS = {
    'APP_NAME': 'My App',
    # Used for json logging
    'MICROSERVICE_NAME: 'myapp',
    # dev, stage, prod ...
    'ENVIRONMENT': ENVIRONMENT,
    'INITIAL_FIXTURES': [
        os.path.join(BASE_DIR, 'testapp', 'fixtures', 'users.json'),
    ],
    'SERVER_IP': '192.168.1.4',
    'REQUEST_ID_HEADER': 'X-Request-ID',
    'RUN_EXTRA_COMMANDS': ['loadinitialusers', 'setup_something'],
    'LOCAL_DJANGO_ADMIN_PASSWORD': os.environ.get('LOCAL_DJANGO_ADMIN_PASSWORD', 'admin'),
    # Only in case you need to change the defaults
    'ENV_THEMES': {
        'local.dev': {
            'title_color': '#000000',
            'css_header_background_color': '#ffffff',
            'env_color': '#00cb38',
            'css_header_text_color': '#000000',
            'css_header_link_color': '#000000',
            'css_header_link_hover_color': '#1e00ac',
            'css_module_background_color': '#ababab',
            'css_module_background_selected_color': '#e3e3e3',
            'css_module_text_color': '#000000',
            'css_module_link_color': '#000000',
            'css_module_link_hover_color': '#3255fe',
            'css_generic_link_color': '#000000',
            'css_save_button_background_color': '#6d6d6d',
            'css_save_button_background_hover_color': '#4a4a4a',
        },
        'dev': {
            'title_color': '#ffffff',
            'env_color': '#00cb68',
            'css_header_background_color': '#007b3b',
            'css_header_text_color': '#f5dd5d',
            'css_header_link_color': '#ffffff',
            'css_header_link_hover_color': '#1e00ac',
            'css_module_background_color': '#006731',
            'css_module_background_selected_color': '#ffffff',
            'css_module_text_color': '#ffffff',
            'css_module_link_color': '#ffffff',
            'css_module_link_hover_color': '#5f0000',
            'css_generic_link_color': '#000000',
            'css_save_button_background_color': '#006731',
            'css_save_button_background_hover_color': '#01a74f',
        },
        'stage': {
            'title_color': '#ffffff',
            'env_color': '#ffcb38',
            'css_header_background_color': '#ff9722',
            'css_header_text_color': '#ffffff',
            'css_header_link_color': '#ffffff',
            'css_header_link_hover_color': '#41aad1',
            'css_module_background_color': '#ca6a00',
            'css_module_background_selected_color': '#ffffff',
            'css_module_text_color': '#ffffff',
            'css_module_link_color': '#ffffff',
            'css_module_link_hover_color': '#41aad1',
            'css_generic_link_color': '#000000',
            'css_save_button_background_color': '#ca6a00',
            'css_save_button_background_hover_color': '#ff9722',
        },
        'playground': {
            'title_color': '#ffffff',
            'env_color': '#00cb38',
            'css_header_background_color': '#09137a',
            'css_header_text_color': '#ffffff',
            'css_header_link_color': '#ffffff',
            'css_header_link_hover_color': '#1e00ac',
            'css_module_background_color': '#0020bf',
            'css_module_background_selected_color': '#e3e3e3',
            'css_module_text_color': '#ffffff',
            'css_module_link_color': '#ffffff',
            'css_module_link_hover_color': '#69c2cc',
            'css_generic_link_color': '#000000',
            'css_save_button_background_color': '#0038ff',
            'css_save_button_background_hover_color': '#02208b',
        },
        'prod': {
            'title_color': '#ffffff',
            'env_color': '#00cb38',
            'css_header_background_color': '#720606',
            'css_header_text_color': '#ffffff',
            'css_header_link_color': '#ffffff',
            'css_header_link_hover_color': '#1e00ac',
            'css_module_background_color': '#e73f41',
            'css_module_background_selected_color': '#e3e3e3',
            'css_module_text_color': '#ffffff',
            'css_module_link_color': '#ffffff',
            'css_module_link_hover_color': '#5f0000',
            'css_generic_link_color': '#000000',
            'css_save_button_background_color': '#720606',
            'css_save_button_background_hover_color': '#4a4a4a',
            # APP_NAME will be replaced by the correct app name set in payla utils settings
            'title': 'APP_NAME',
        },
    }
}

Development of this project

Please install hatch as this is the tool we use for releasing.

To run tests:

hatch run test-local

To format code:

hatch run format

Django pytest integration docs.