Skip to main content

nqlog

Luca de Alfaro, 2020

Not quite the logger Google wrote, not quite the logger Luca wanted, but at least:

  • It logs to the Google cloud.
  • It groups logs for the same "request" into the same log entry.
  • The severity of the grouped lines is equal to the maximum severity of any line in the group.
  • It avoids running string formatting for strings that are logged below the minimum level.

This seems trivial and normal, but unfortunately, it is no longer the standard behavior. In GC, log lines are no longer automatically grouped together for each request, and even if you figure out how to have them grouped, the group no longer inherits the maximum severity of any line in the group.

The chief limitation of this logger is that it works only for single-threaded services. Extension to multi-threaded services is a work in progress.

Use

Keys to GC

In your "main" file, you need:

import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "keys/google_cloud_keys.json"

where of course you replace the path with the actual one. The keys are json keys to a service account that has logging rights on Google cloud.

Logger use

All you need is:

from nqlog import logthis, logging

Then, you need to decorate the functions whose execution corresponds to a "request". This can be the execution of a web request (e.g., a bottle entry point), or the execution of a task from a scheduler.

@logthis()
def function_to_be_logged():
    ...

Once this is done, whenever inside the function to be logged, or one of the things it calls, you do:

logging.info("This is a log line")

the log line is logged together with all other lines of the request. The grouping is thread-safe, so if you call function_to_be_logged from multiple threads (e.g., to service requests in parallel), everything works as it should.

The logthis decorator takes optional parameters:

Decorator to cause a function to be logged.
:param log_name: Name of the log to be used for logging.
:param resource: resource to be used for logging.  You can create a
    resource e.g. via:
    from google.cloud.logging.resource import Resource
    resource = Resource( type='gae_app',
                labels={
                    "project_id": os.getenv("GOOGLE_CLOUD_PROJECT")
                    "module_id": os.getenv("GAE_MODULE")
                    "service_id": os.getenv("GAE_SERVICE")
                    "version_id": os.getenv("GAE_VERSION")
                   })
:param max_duration: Max duration of a log entry in seconds.
    If the request takes longer than this, the log entry is broken into
    multiple log entries, each of at most max_duration length.
:param level: Minimum level at which logging occurs.

Efficiency

Often, building a log line is computationally intensive, due to the process of converting data types and structures to a string. nqlog contains an optimization. If instead of doing:

logging.debug("The value of my monster data structure is: %r" % monster)

you do:

logging.debug("The value of my monster data structure is: %r", monster)

with a , instead of a %, then nqlog avoids converting monster to its representation unless the log level is DEBUG or lower.

Metadata

Release files for nqlog 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nqlog 0.1.0
File Size Uploaded Provenance Actions
nqlog-0.1.0.tar.gz 6.1 kB Without provenance Details for nqlog-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nqlog 0.1.0
File Interpreter ABI Platform Actions
nqlog-0.1.0-py3-none-any.whl Python 3 none any Details for nqlog-0.1.0-py3-none-any.whl

Total release size: 12.6 kB

Release files / nqlog-0.1.0.tar.gz

Download URL nqlog-0.1.0.tar.gz
Size 6.1 kB
Tags Source
SHA-256 checksum
How to use checksums
2b8a015351d8e07bd77229c8c005de6053a2e14fe18e65f322fcd3cbe8a60239
BLAKE2b-256 checksum
How to use checksums
56e952ccfd5e6a761cf29b4f198953d36ab42fbd8b1077191db2ee3295898692
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.7

Release files / nqlog-0.1.0-py3-none-any.whl

Download URL nqlog-0.1.0-py3-none-any.whl
Size 6.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e72736ab6d1a2ea87e67302e001d81d4fa2b1884a07e2d83a5792c70af437d1c
BLAKE2b-256 checksum
How to use checksums
f5f94320b6cc78fcc4bc7f424b7f0becaf041638c533e74a28b9ff2ecabf130d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.7

Security for nqlog 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page