formenergy-observability 0.3.2

OTel tracing/logging tools, emphasizing data pipeline use cases with Dagster.

Form Energy Observability Python Library

OTel tracing and logging tools, emphasizing data pipeline use cases with Dagster.

Using the Library

Install from PyPI

pip install formenergy-observability

Features

See the examples/ directory for examples you can run and screenshots of the traces they produce.

Some key features demonstrated in the examples:

Context-aware tracing, so that spans and events can inherit attributes from parents:

from form_observability import ContextAwareTracer, ctx

_trace = ContextAwareTracer(__name__)

with ctx.set({"my_attribute_name": my_attribute_value}):
    # There could be many intervening function calls and spans.
    # Spans and events in this context automatically have the my_attribute_name
    # attribute set on them.
    _trace.add_event("my_event")

Trace context propagation for Dagster ops:

from form_observability.dagster_otel import otel_op, publish_current_trace_context
from form_observability import ContextAwareTracer

_trace = ContextAwareTracer(__name__)

@op
def root_trace_context_op(context):
    with _trace.start_as_current_span("root"):
        # writes the trace context (as an asset materialization event) for later ops
        publish_current_trace_context(context)

@otel_op()  # picks up the trace context and starts a new span for this op
def my_op(context):
    _trace.add_event("my_event")

@graph
def my_graph()
    # In the graph, publish a root trace using a regular op, and then subsequent
    # @otel_op ops will pick it up and contribute to the same trace.
    setup_done = root_trace_context_op()
    my_op(start=setup_done)

Running the Examples

Get an API key for uploading traces to your observability service (preferably in a development environment), set it in your environment, and run the examples. This should publish traces that you can then view.

export OTEL_EXPORTER_OTLP_HEADERS_JSON="{'x-honeycomb-team': '<API key>'}"
export OTEL_EXPORTER_OTLP_ENDPOINT="your observability service, defaults to Honeycomb"
python examples/basics.py
dagster job execute --python-file examples/dagster_job.py

License

This library is provided under the MIT license, see LICENSE.

Release Notes

• 0.3.2 Update grpcio minimum dependency version to 1.48.1 and dagster to 1.5.0. Thanks @dang-kaiko !
• 0.3.1 Fix type used for otlp_headers to be a dict. Thanks @spenczar !
• 0.3.0 Add ContextAwareTracer.start_new_linked_trace.
• 0.2.0 Switch to context.op_handle and materialization.metadata as dict for Dagster 1.2.
• 0.1.0 Initial external release, including ContextAwareTracer, ctx, and @otel_op.

Development

If you contribute, please ensure you and your employer accept the license for any of your contributions.

Setup

Create and activate a virtual environment with your favorite tool, such as direnv. This project requires Python 3.8.

pip install flit
flit install --deps all --symlink [--python /path/to/venv/python]

By default, flit installs into the current virtual environment. Using the --python argument can let you install into a specific venv other than the currently active one. For example you can activate some other venv, run which python, and then pass the output of that command to flit's --python argument when installing from the form-observability repo.

Initialize pre-commit hooks:

pre-commit install --hook-type pre-commit --hook-type pre-push

Running Tests

From the project root directory:

pytest