python-logging-loki

Python logging handler for Loki.
https://grafana.com/loki

Installation

pip install inuits-python-logging-loki

Usage

import logging
import logging_loki


handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push",
    tags={"application": "my-app"},
    headers={"X-Scope-OrgID": "example-id"},
    auth=("username", "password"),
    props_to_labels = ["foo"]
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(
    "Something happened",
    extra={"tags": {"service": "my-service"}},
)

Example above will send Something happened message along with these labels:

• Default labels from handler
• Message level as serverity
• Logger's name as logger
• Labels from tags item of extra dict
• Property foo from log record will be sent as loki label

Properties to label

Using a dict instead of a list for props_to_labels will enable renaming labels

handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push",
    tags={"application": "my-app"},
    props_to_labels = {
        "otelTraceID": "trace_id"
        "otelSpanID":  "span_id"
    }
)

In this case, the properties otelTraceID & otelSpanID will be renamed to trace_id & span_id loki labels

Non-blocking mode

Adding a LokiHandler to a logger directly is blocking: every log call does an HTTP POST to Loki on the calling thread and waits for the response. In a request handler that puts Loki's latency (and Loki's rate limiting) straight into your own response time.

Use LokiQueueHandler instead. It attaches to the logger, creates the LokiHandler and a QueueListener, and starts the listener, so the calling thread only puts the record on a queue.

import logging_loki
from queue import Queue


handler = logging_loki.LokiQueueHandler(
    Queue(-1),
    batch_interval=2,  # seconds; omit or pass 0 to push every record separately
    url="https://my-loki-instance/loki/api/v1/push",
    tags={"application": "my-app"},
    headers={"X-Scope-OrgID": "example-id"},
    auth=("username", "password"),
    props_to_labels=["foo"],
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(...)

If you wire the queue up by hand, you must start the listener yourself — a QueueListener that is never started means nothing ever drains the queue and no log line reaches Loki:

queue = Queue(-1)
handler = logging.handlers.QueueHandler(queue)
handler_loki = logging_loki.LokiHandler(url=..., tags=...)
listener = logging.handlers.QueueListener(queue, handler_loki)
listener.start()  # <- required

Batching and timeouts

With batch_interval set, records are buffered and pushed as one request. This is what keeps Loki from answering with 429 Too Many Requests: without it every log line is its own POST and its own stream.

A push that Loki rejects (429, or any other unexpected status) raises logging_loki.emitter.LokiPushError and the records in that batch are dropped. The HTTP session is kept open, since the connection is still healthy — only a transport-level failure closes it and forces a reconnect.

Forking servers

LokiQueueHandler does its work on background threads, and threads do not survive fork(). Handlers built before the fork — a logger created at import time under gunicorn --preload, for example — recreate their queue, listener and flush threads in the child automatically, so each worker delivers its own records and does not re-send the parent's.