Code examples

Authoritative code examples sources for non-logging utilities.

Why?

logging.config is more often than not hardcoded within a package’s source code. Requiring editting of source code to adjust logging configuration is a constrictive design.

Remove logging.config from the source code instead rely on a, runtime strictly validated against a schema, YAML config file. The If not found, YAML config is exported otherwise it’s not.

When a logging is not shown for a particular package, open the exported logging.config YAML file. Add the offending package to the loggers section or if already there, increase the logging level.

For example, for asyncio, adjust logging level from logging.WARNING –> logging.ERROR

Bye bye disruptive informational logging warning messages.

logging_strict comes with a logging.config yaml validator. So can separately check YAML file.

On app|worker restart, the logging configuration changes take effect.

Exporting – when

Exports occur before the logging.config yaml files are needed. There are two process types: worker and app

When an app is run, it exports the app logging configuration.

Right before a ProcessPool runs, it exports the worker logging configuration.

Right before a thread or ThreadPool runs, G’d and Darwin sit down to decide which calamity will befall you. Best to avoid that cuz Python logging module is thread-safe. Changes to the logging.config in one thread affects them all and those changes last as long as the app runs.

Safe means safe to remove you from the gene pool. Would be a great name for a horror movie. Don’t be in that movie.

Exporting – where/what

Export location (on linux): $HOME/.local/share/[package name]/

This is xdg user data dir and the configuration is per package. Python logging configurations’ cascade!

Whats exported?

• one for the app

• At least one, for the multiprocessing workers

If a user|coder edits and makes a change, undo’ing those changes would be considered quite rude, minimally, poor ettiquette.

So that gauntlets stay on and package authors live long fulfilling peaceful uneventful lives, overwrite existing logging config yaml files never happens. Although fully capable, just absolutely refuses to do so!

If confident no changes have been made, can manually delete (unlink).

There will be no need for gauntlets, can safely put those away.

Upgrade path

How to upgrade a particular logging.config yaml file?

Best to increment the version and switch the code base to use the latest version

Custom changes should be upstreamed.

Preferred the previous version

There currently isn’t a means to change which logging.config yaml file a package uses.

This sounds like a job for user preference database, gschema. Not yet implemented

Validation

logging.handlers, each, expects parameters to have the correct data type.

yaml package strictyaml, default data type is str, for other types, the function variable name and type must be known (and supported) beforehand.

For custom (handlers, filters, and formatters) functions, there is no way to know beforehand the parameter name and therefore the data type, parameter type will become str

(Assuming the virtual environment is activated)

Within source code (tree)

logging_strict_validate_yaml

Processed: 4 / 4
Success / fail: 4 / 0
last (3): ~/Downloads/logging_strict/src/logging_strict/configs/mp_1_asz.worker.logging.config.yaml

logging_strict_validate_yaml --category worker

Processed: 3 / 3
Success / fail: 3 / 0
last (2): ~/Downloads/logging_strict/src/logging_strict/configs/mp_1_asz.worker.logging.config.yaml

logging_strict_validate_yaml --category app

Processed: 1 / 1
Success / fail: 1 / 0
last (0): ~/Downloads/logging_strict/src/logging_strict/configs/textual_1_asz.app.logging.config.yaml

logging_strict_validate_yaml $HOME/.local/share/logging_strict/ worker

logging_strict_validate_yaml $HOME/.local/share/logging_strict/ app

pre-commit

Locally

repos:
  - repo: local
    hooks:
      - id: validate-logging-config-yaml
        name: validate-logging-config-yaml
        entry: logging_strict_validate_yaml
        language: python
        require_serial: true
        pass_filenames: false

Normal usage

repos:
  - repo: https://github.com/msftcangoblowm/logging-strict
    rev: 0.1.0
    hooks:
      - id: validate-logging-config-yaml
        name: validate-logging-config-yaml
        entry: logging_strict_validate_yaml
        language: python
        require_serial: true
       pass_filenames: false

install

You know how to use pip. This is not that.

Lets discuss integrating logging-strict into your app and history dust binning hardcoded logging configuration.

def _process_args(): ...

def main():
    d_out = _process_args()
    ...
    # app logging config stuff <--- here!
    app = MyApp()  # <-- not within here
    ...

if __name__ = "__main__":
    main()

from logging_strict.constants import
from logging_strict import ui_yaml_curated, LoggingState

genre = "textual"
version_no = "1"
flavor = "asz"  # < -- Yet unpublished testing UI package
package_start_relative_folder = ""

LoggingState().is_state_app = True
ui_yaml_curated(
    genre,
    flavor,
    version_no=version_no,
    package_start_relative_folder=package_start_relative_folder,  # <-- narrows the search
)

from mypackage.constants import urpackagename, package_data_folder_start
from logging_strict import setup_ui_other, LoggingState

genre = "textual"
flavor = "asz"  # < -- Yet unpublished testing UI package
version_no = "1"
package_start_relative_folder = ""

LoggingState().is_state_app = True
setup_ui_other(
    urpackagename,  # <-- Would have been better to curate within logging_strict
    package_data_folder_start,
    genre,
    flavor,
    version_no=version_no,
    package_start_relative_folder=package_start_relative_folder,
)

worker

This is a 2 step process.

• Step 1 – entrypoint

  Extracts yaml from package, validates, then passes as str to the worker process

• Step 2 – worker

  yaml str –> logging.config.dictConfig

from logging_strict import worker_yaml_curated

genre = "mp"
flavor = "asz"

str_yaml = worker_yaml_curated(genre, flavor)

from logging_strict import worker_yaml_curated

package = "someotherpackage"
package_data_folder_start = "data"  # differs so need to check this folder name

genre = "mp"
flavor = "asz"

str_yaml = setup_worker_other(package, package_data_folder_start, genre, flavor)

from logging_strict import setup_logging_yaml

setup_logging_yaml(str_yaml)

Public API

from logging_strict import (
   LoggingConfigCategory,
   LoggingState,
   LoggingYamlType,
   setup_ui_other,
   ui_yaml_curated,
   setup_worker_other,
   worker_yaml_curated,
   setup_logging_yaml,
   LoggingStrictError,
   LoggingStrictPackageNameRequired,
   LoggingStrictPackageStartFolderNameRequired,
   LoggingStrictProcessCategoryRequired,
   LoggingStrictGenreRequired,
)

• LoggingConfigCategory

  tl;dr; ^^ won’t need this ^^

  Process categories Enum. Iterate over the Enum values, using class method, categories.

  strict_logging public methods are convenience functions for class, strict_logging.logging_api.LoggingConfigYaml. If LoggingConfigYaml used directly, choose one of the LoggingConfigCategory values to pass as param, category.

• LoggingYamlType

  tl;dr; ^^ won’t need this ^^

  Useful only during strict type checking. class LoggingConfigYaml implements LoggingYamlType interface and is a direct subclass

• LoggingStrictError

  logging_strict catch all Exception. Base type of other exceptions. Implements ValueError

  The other exceptions are self explanatory. When creating worker entrypoints, can set exit codes based on which exception occurred.

Whats strictyaml?

Unfortunately yaml spec is too broad, allowing undesirable complexity, which are a frequent cause of security issues. Read more:

• [why]

• [why nots]

strictyaml ([docs]) mitigates yaml security issues:

• by only supporting a subset of the yaml spec

• type-safe YAML parsing and validation against a schema

  In our case, specialized to support the built-in Python logging.handlers and adaptable enough to support custom handlers, filters, and formatters