nqlog
Luca de Alfaro, 2020
Not quite the logger Google wrote, not quite the logger Luca wanted, but at least:
- It logs to the Google cloud.
- It groups logs for the same "request" into the same log entry.
- The severity of the grouped lines is equal to the maximum severity of any line in the group.
- It avoids running string formatting for strings that are logged below the minimum level.
This seems trivial and normal, but unfortunately, it is no longer the standard behavior. In GC, log lines are no longer automatically grouped together for each request, and even if you figure out how to have them grouped, the group no longer inherits the maximum severity of any line in the group.
The chief limitation of this logger is that it works only for single-threaded services. Extension to multi-threaded services is a work in progress.
Use
Keys to GC
In your "main" file, you need:
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "keys/google_cloud_keys.json"
where of course you replace the path with the actual one. The keys are json keys to a service account that has logging rights on Google cloud.
Logger use
All you need is:
from nqlog import logthis, logging
Then, you need to decorate the functions whose execution corresponds to a "request". This can be the execution of a web request (e.g., a bottle entry point), or the execution of a task from a scheduler.
@logthis()
def function_to_be_logged():
...
Once this is done, whenever inside the function to be logged, or one of the things it calls, you do:
logging.info("This is a log line")
the log line is logged together with all other lines of the request. The grouping is thread-safe, so if you call function_to_be_logged from multiple threads (e.g., to service requests in parallel), everything works as it should.
The logthis decorator takes optional parameters:
Decorator to cause a function to be logged.
:param log_name: Name of the log to be used for logging.
:param resource: resource to be used for logging. You can create a
resource e.g. via:
from google.cloud.logging.resource import Resource
resource = Resource( type='gae_app',
labels={
"project_id": os.getenv("GOOGLE_CLOUD_PROJECT")
"module_id": os.getenv("GAE_MODULE")
"service_id": os.getenv("GAE_SERVICE")
"version_id": os.getenv("GAE_VERSION")
})
:param max_duration: Max duration of a log entry in seconds.
If the request takes longer than this, the log entry is broken into
multiple log entries, each of at most max_duration length.
:param level: Minimum level at which logging occurs.
Efficiency
Often, building a log line is computationally intensive, due to the process of converting data types and structures to a string. nqlog contains an optimization. If instead of doing:
logging.debug("The value of my monster data structure is: %r" % monster)
you do:
logging.debug("The value of my monster data structure is: %r", monster)
with a , instead of a %, then nqlog avoids converting monster to its representation unless the log level is DEBUG or lower.
Metadata
Release files for nqlog 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nqlog-0.1.0.tar.gz | 6.1 kB | Without provenance | Details for nqlog-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nqlog-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for nqlog-0.1.0-py3-none-any.whl |
Total release size: 12.6 kB
Release files / nqlog-0.1.0.tar.gz
| Download URL | nqlog-0.1.0.tar.gz |
|---|---|
| Size | 6.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2b8a015351d8e07bd77229c8c005de6053a2e14fe18e65f322fcd3cbe8a60239
|
|
BLAKE2b-256 checksum How to use checksums |
56e952ccfd5e6a761cf29b4f198953d36ab42fbd8b1077191db2ee3295898692
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.7
|
Release files / nqlog-0.1.0-py3-none-any.whl
| Download URL | nqlog-0.1.0-py3-none-any.whl |
|---|---|
| Size | 6.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e72736ab6d1a2ea87e67302e001d81d4fa2b1884a07e2d83a5792c70af437d1c
|
|
BLAKE2b-256 checksum How to use checksums |
f5f94320b6cc78fcc4bc7f424b7f0becaf041638c533e74a28b9ff2ecabf130d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.7
|
Security for nqlog 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.