a wrapper around `opentelemetry` and `opentelemetry-instrumentation-*` to make life a bit easier
Project description
opentelemetry_wrapper
a wrapper around opentelemetry
and opentelemetry-instrumentation-*
to make life a bit easier
design principles
safe
- never crash the application
- fallback to non-
opentelemetry
if necessary - ignore input over raising an exception
- possible exception: it may be better to fail at startup than to run with known bad config
- make dangerous things impossible (by design) > difficult (guardrails) > unusual (defaults) > understood (docs)
- there should be a "best practice" somewhere in that statement
- fallback to non-
- hard to get wrong
- idempotent, even when you instrument the same thing in different ways from different places
- "be conservative in what you send, be liberal in what you accept"
- note: easy and simple mean different things
unsurprising
- simple, idiomatic, succinct, and pretty
- decorators over wrappers
- wrappers over context managers
- context mangers over
manually managing spansanything else (note: this is still a todo)
- reasonable documented defaults
- magic may be hard to understand, but it's better than being irritating
- emits zero logs/spans (of its own) at runtime
- fail silently over flailing noisily
- drowning out real logs can be worse than being useless (e.g., you could crash
fluentd
- ask me how I know)
helpful
- machine-readable first, human-readable a close second
- no newlines in logs or spans
- no whitespace-delimited ambiguity
- json all the things (within reason)
- provide all available application context
- we want to know all we can about what's going on and where
- code introspection and runtime analysis if we can make it fast enough
usage
TL;DR:
- call
instrument_all()
to instrumentlogging
andrequests
- instrument your FastAPI app using
instrument_fastapi_app(FastAPI(...))
- use
@instrument_decorate
on any function or class you want to monitor- set
OTEL_WRAPPER_DISABLED=true
in your CICD tests, especially if you're usingpytest
env vars
Variable Name | Description | Default (if not set) |
---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
Looks like http://tempo.localhost:4317 . |
NA (traces are not exported to any OTLP endpoint) |
OTEL_EXPORTER_OTLP_HEADER |
Looks like Header-Name=header value , where values can contain space ('\x20'). To insert multiple headers, delimit by any other whitespace char. |
NA (no header sent to OTLP endpoint) |
OTEL_EXPORTER_OTLP_INSECURE |
Set to true to disable SSL for OTLP trace exports, or false to always verify. |
NA (follows OpenTelemetry default, which is secure for https and insecure for http) |
OTEL_EXPORTER_PROMETHEUS_PORT |
The port on which to expose metrics for Prometheus, running in parallel as a WSGI app. (E.g. 9464 to expose http://localhost:9464/* ) WARNING: do not use the same port as your app. |
NA (no Prometheus server) |
OTEL_EXPORTER_PROMETHEUS_ENDPOINT |
An endpoint on which to expose metrics for Prometheus via FastAPI. (E.g. /metrics ) WARNING: this can clash with your fastapi routes. |
/metrics (set to a space to avoid creating a Prometheus endpoint) |
OTEL_HEADER_ATTRIBUTES |
List of HTTP headers to extract from incoming requests as span attributes, split by comma. | x-userinfo |
OTEL_LOG_LEVEL |
Log level used by the logging instrumentor (case-insensitive). | info |
OTEL_SERVICE_NAME |
Sets the value of the service.name resource attribute. |
f'{k8s namespace}/{k8s deployment}/{k8s pod}' or f'{username}@{hostname}.{domain}:<{filename of main}>' |
OTEL_SERVICE_NAMESPACE |
Sets the value of the service.namespace resource attribute. |
f'{k8s namespace}' or None |
OTEL_WRAPPER_DISABLED |
Set to true to disable tracing globally (e.g. when running pytest). |
false (tracing is enabled) |
Note:
The
service.name
andservice.namespace
can also be set viaOTEL_RESOURCE_ATTRIBUTES
, but any settings there will be overwritten byOTEL_SERVICE_NAME
andOTEL_SERVICE_NAMESPACE
. For more details, read the original OpenTelemetry docsIf you don't have a
serviceAccount
mounted in your pod (i.e.automountServiceAccountToken: false
), then you should manually setOTEL_SERVICE_NAMESPACE
using the following yaml:env: - name: OTEL_SERVICE_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace
@instrument_decorate
decorator for functions and classes
- decorating a function creates a span whenever the function is called
- the span name is set to the function or class name, and attributes are added for the filename, line number, etc
- also works for async functions
- also works for classes and dataclasses
- creates a span for every method call, including init (or post_init), new, and call
- creates a span for every property get, set, or delete
- if this is too verbose, feel free to decorate specific methods in the class instead
- it is not recommended to use this on functions that are called thousands of times or classes you create thousands of instances for (e.g. pydantic classes, usually), since it can be excessively noisy
from opentelemetry_wrapper import instrument_decorate
@instrument_decorate
def square(x):
return x * x
@instrument_decorate
async def cube(x):
return x * x * x
@instrument_decorate
class Thing:
def __init__(self):
self.x = 1
instrumenting the builtin logging
module
- sets a root logger handler (or more than one) that can output logs to the console or to a file path
- logs are output as json by default to make them easier to work with downstream, set
print_json=False
to disable- the
verbose
parameter has no effect on json output, which is maximally verbose - otherwise, it defaults to
True
- set toFalse
to print less text to the console
- the
- the log level can be specified via the
level
arg, but defaults to whatever was set via theOTEL_LOG_LEVEL
env var
import logging
from opentelemetry_wrapper import instrument_logging
instrument_logging()
logging.warning('...')
instrumenting fastapi
- Warning: using
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor
does not work as intended - use
instrument_fastapi_app
, for example:
from fastapi import FastAPI
from opentelemetry_wrapper import instrument_fastapi_app
app = instrument_fastapi_app(FastAPI(...))
Capturing headers:
- to avoid capturing any headers as spans, set a blank header, like
OTEL_HEADER_ATTRIBUTES=,
OTEL_HEADER_ATTRIBUTES
will attempt to parse base64 and json, and will flatten the data structureOTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST
behaves similarly, but does not flatten the structure
prometheus
- set (for example)
OTEL_EXPORTER_PROMETHEUS_ENDPOINT=/metrics
- remember to whitelist
/metrics
in OPA - repeated and trailing slashes are normalized (e.g.
///metrics/
->/metrics
) - if you set
OTEL_EXPORTER_PROMETHEUS_ENDPOINT=/metrics/
, a redirect will be set up from/metrics
to/metrics/
- if you set the port instead, the created wsgi app accepts any endpoint
- if you set up both, the endpoint will be created in fastapi and a separate wsgi app will also be created
- the
/graph
endpoint is not available, you'll need to actually run prometheus to get that
features
- Make instrumentation (more) idempotent:
- you can call the instrument functions unlimited times from multiple places in your codebase, and it'll work the same
- e.g., a class definition, a defined method of the class, a class instance, and a method from the instance
- Make re-instrumentation of
logging
actually work when passing in a new format string - Make
logging
print as a one-line JSON dict by default, with a lot of magic to convert stuff to valid json - logs and spans contain info about which thread / process and which file / function / line of code it came from
- and the k8s namespace and pod, if applicable, otherwise the local pc name
- Provide support for decorating functions and classes
- Provide support for instrumentation of dataclasses
- NOTE: Global instrumentation needs to be run before any dataclasses are initialized
- Otherwise, use the decorator on each class as usual (since it is idempotent anyway)
- Add global instrumentation of FastAPI
- sometimes works even after apps are created for some reason, likely due to how Uvicorn runs in a new process
- but somehow sometimes doesn't work in prod, for equally unknown reasons
- probably best to instrument each app instance
- Logs OIDC http headers as span attributes for FastAPI
- Creates OTLP exporters if specific env vars (below) are set
- Pushes logs, metrics, and traces to the OTEL endpoint, if configured
- Note: only logs and traces are printed to console, metrics are too noisy
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for opentelemetry_wrapper-0.1.17.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a80a7ac48ed5cc3643f3ac1292324ecd9f0921bd0c03ff34aaa8ebf04edde64 |
|
MD5 | 7d5c8b76c47e3393d29ed2070fe5b93c |
|
BLAKE2b-256 | 96c93ed5eac201177c42a217ba12336fae62f6e4ec166e8bf8002514883a0f93 |
Close
Hashes for opentelemetry_wrapper-0.1.17-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ac9a7167deba3aa7510db659bf09a389922c99526193bb93b30c9215abedc48 |
|
MD5 | ec6054159d8adbda8f64715e0a9318f0 |
|
BLAKE2b-256 | ad796c2b4f4a2a94646fbcc89f790dee764d41dce8034574d3a504b83c1b91da |