Skip to main content

ONDEWO Logo

ONDEWO Logging

This is the logging package for ONDEWO products. It allows for easy integration with our EFK stack, and adds some useful features to the base python logging package (such as timing and exception handling), and handles GRPC error messages nicely.

Useage

To use this library, first pip install it:

pip install ondewo-logging

then import it into your project like so:

from ondewo.logging.logger import logger_console

Note

In order for logger to log module_name, docker_image_name, and git_repo_name one has to pass them as environment variables to the container where the logged service is running.

Decorators

A couple of decorators are included:

from ondewo.logging.decorators import Timer, timing, exception_handling, exception_silencing

The Timer class can be used as a context manager:

with Timer() as t:
  sleep(1)

or as a decorator:

@Timer()
def sleeptime():
  sleep(1)

and can be used with different messages or logging levels:

  • Logging level: @Timer(logger=logger_console.info)
  • Message: @Timer(message="MESSAGE WITH TIME {} {}"), @Timer(message="SIMPLER MESSAGE WITHOUT TIME")
  • Disable argument logging: @Timer(log_arguments=False)
  • Enable exception suppression: @Timer(supress_exceptions=True)

See the tests for detailed examples of how these work.

Timing is just an instance of the Timer class:

timing = Timer()

for backwards compatibility.

The exception_handling function is a decorator which will log errors nicely using the ondewo logging syntax (below). It will also log the inputs and outputs of the function. The exception_silencing function just shows the inputs and outputs and gets rid of the stacktrace, it can be useful for debugging. Finally, log_arguments will dump the inputs and outputs of a function into the logs.

Ondewo log format

The structure of the logs looks like this:

message: Dict[str, Any] = {
  "message": f"Here is the normal log, including relevant information such the magic number: {magic number}. These values are also added seperately below, either just with the variable name or some other relevant name. Finally, there are some tags to help with searching through the logs.",
  "magic_number": magic_number,
  "tags": ["magic", "number"]
}

Note on tags:

The tags allow for easy searching and grouping in kibana. They can be added in a somewhat ad-hoc manner by the programmer on the ground, though some (like 'timing') are standardised. Please talk to your project team lead for details.

Fluentd

Quickstart

  1. git clone https://github.com/ondewo/ondewo-logging-python
  2. make
  3. edit the fluentd config with the url and password of your elasticsearch host:
sed -i 's/<PASSWORD>/my_password/' './fluentd/conf/fluent.conf'
sed -i 's/<HOST>/my_elasticsearch_host/' './fluentd/conf/fluent.conf'
  1. run fluentd docker-compose -f fluentd/docker-compose.yaml up -d

You now have a fluentd message handler running on your machine. If you use the ondewo.logging library, your logs will be shipped to your elasticsearch server.

Fluentd Config

Per the fluentd/docker-compose.yaml, we map the configuration files and the logs into the fluentd image and open some ports. We also need to chown -R 100:"$GID" fluentd/log. That command should allow both you and fluentd to read the logs.

Beyond that, it is just a question of formatting the logs wherever they come from. Here is the example from the fluentd config that sends stuff to the fluentd stdout, so you can see the logs from all your images in the same place.

<source>
  @type forward
  port 24224
</source>

# py.console logging gets piped to stdout
<match py.console.**>
  @type stdout
  <format>
      @type ltsv
      delimiter_pattern :
      label_delimiter =
  </format>
</match>

In this conf, we recieve imput over a tcp connection, then dumps the output to stdout, so you can use that stream to watch log output via fluentd. The config is also set up to save all the logs locally, and ship them to a remote server.

Automatic Release Process

The entire process is automated to make development easier. The actual steps are simple:

TODOs in Pull Request before the release:

  • Update the Version number inside the Makefile

    • ! : Major and Minor Version Number must be the same for Client and API at all times

      example: API 2.9.0 --> Client 2.9.X

  • Check if RELEASE.md is up-to-date

  • Update the Version number inside the setup.py by using:

    make update_setup
    

TODOs after Pull Request was merged in:

  • Checkout master:
    git checkout master
    
  • Pull the new stuff:
    git pull
    
  • Release:
    make ondewo_release
    

The make ondewo_release command can be divided into 5 steps:

  • cloning the devops-accounts repository and extracting the credentials
  • creating and pushing the release branch
  • creating and pushing the release tag
  • creating the GitHub release
  • creating and pushing the new PyPi release

The variables for the GitHub Access Token, PyPi Username and Password are all inside of the Makefile, but the values are overwritten during make ondewo_release, because they are passed from the devops-accounts repo as arguments to the actual release command.

Metadata

Release files for ondewo-logging 3.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ondewo-logging 3.2.3
File Size Uploaded Provenance Actions
ondewo-logging-3.2.3.tar.gz 12.4 kB Without provenance Details for ondewo-logging-3.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ondewo-logging 3.2.3
File Interpreter ABI Platform Actions
ondewo_logging-3.2.3-py2.py3-none-any.whl Python 3, Python 2 none any Details for ondewo_logging-3.2.3-py2.py3-none-any.whl

Total release size: 25.1 kB

Release files / ondewo-logging-3.2.3.tar.gz

Download URL ondewo-logging-3.2.3.tar.gz
Size 12.4 kB
Tags Source
SHA-256 checksum
How to use checksums
57025ebbe2c4d070c6452c6b62cc7456f4f1558d20c74c6883701c0a82808d90
BLAKE2b-256 checksum
How to use checksums
e084a4ea8633d5c419a71a245b842b112895ac7221f0c32b6dbeced163b2e721
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.13

Release files / ondewo_logging-3.2.3-py2.py3-none-any.whl

Download URL ondewo_logging-3.2.3-py2.py3-none-any.whl
Size 12.6 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
c09748075cd61dcbe0d7b4a14115d3729447fc530b9801c7595d779dcd7b65d2
BLAKE2b-256 checksum
How to use checksums
03e0cbe19f464acd8756911ede2b293526a09287ad0acfbca2818f57222291eb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.13

Security for ondewo-logging 3.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page