OTel tracing/logging tools, emphasizing data pipeline use cases with Dagster.
Project description
Form Energy Observability Python Library
OTel tracing and logging tools, emphasizing data pipeline use cases with Dagster.
Using the Library
Install from PyPI
pip install formenergy-observability
Features
See the examples/ directory for examples you can run and screenshots of the traces they produce.
Some key features demonstrated in the examples:
Context-aware tracing, so that spans and events can inherit attributes from parents:
from form_observability import ContextAwareTracer, ctx
_trace = ContextAwareTracer(__name__)
with ctx.set({"test_id": my_test_id}):
# many intervening function calls and spans
_trace.add_event("my_event") # automatically has the test_id attribute set on it
Trace context propagation for Dagster ops:
from form_observability.dagster_otel import otel_op, publish_current_trace_context
from form_observability import ContextAwareTracer
_trace = ContextAwareTracer(__name__)
@op
def root_trace_context_op(context):
with _trace.start_as_current_span("root"):
# writes the trace context (as an asset materialization event) for later ops
publish_current_trace_context(context)
@otel_op() # picks up the trace context and starts a new span for this op
def my_op(context):
_trace.add_event("my_event")
@graph
def my_graph()
# In the graph, publish a root trace using a regular op, and then subsequent
# @otel_op ops will pick it up and contribute to the same trace.
setup_done = root_trace_context_op()
my_op(start=setup_done)
Running the Examples
Get an API key for uploading traces to your observability service (preferably in a development environment), set it in your environment, and run the examples. This should publish traces that you can then view.
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=<API key>"
export OTEL_EXPORTER_OTLP_ENDPOINT="your observability service, defaults to Honeycomb"
python examples/basics.py
dagster job execute --python-file examples/dagster_job.py
License
This library is provided under the MIT license, see LICENSE.txt.
Release Notes
- 0.1.0 Initial external release, including
ContextAwareTracer
,ctx
, and@otel_op
.
Development
If you contribute, please ensure you and your employer accept the license for any of your contributions.
Setup
Create and activate a virtual environment with your favorite tool, such as direnv. This project requires Python 3.8.
pip install flit
flit install --deps all --symlink [--python /path/to/venv/python]
By default, flit
installs into the current virtual environment. Using the --python
argument can let you install into a specific venv other than the currently active one. For example you can activate some other venv, run which python
, and then pass the output of that command to flit
's --python
argument when installing from the form-observability
repo.
Initialize pre-commit hooks:
pre-commit install --hook-type pre-commit --hook-type pre-push
Running Tests
From the project root directory:
pytest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for formenergy-observability-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6445208831cf31238ccb96ba9f02624d59dfebb0b90a9d1c0e676b4871ecafb3 |
|
MD5 | 10dc63a08a22fe79b5b23da830039a9a |
|
BLAKE2b-256 | cd8d0f4e4e416f54dbd5f1991c80b486f102f801be2a7b4c00a314d14b4357b6 |
Hashes for formenergy_observability-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df068af41cfd96b63035338081dbc80ba8734fd4e2684b1309661d62f2645678 |
|
MD5 | 759e56b0422f2927a2805c8ed701f11a |
|
BLAKE2b-256 | 3275b859f310a90dcb2937f2321cc2599ae8c238416a6272ce9281e9e9f3cd81 |