opentelemetry-instrumentation-dagster

Automatic tracing for Dagster pipelines: pip install, run your pipeline through the opentelemetry-instrument launcher, and every @op/@asset/@multi_asset/@asset_check/@dbt_assets gets a span. No @traced() decorators, no code changes, no imports in your own pipeline files at all.

Built on dagster-otel (same author) -- that project does the actual span creation via an explicit @traced() decorator; this one's only job is applying it automatically instead. See docs/design.md for why they're two separate packages rather than one.

Table of Contents

• Installation
• Quick Start
• Usage
• What's covered
• Configuration
• multiprocess/k8s_job_executor
• Compatibility
• Why a separate package
• Contributing
• Roadmap
• License

Installation

pip install opentelemetry-instrumentation-dagster

Quick Start

docker compose up -d brings up a local Jaeger (no other setup) so you can see a real trace land within a couple of minutes. examples/definitions.py is a plain Dagster job and asset -- zero @traced() calls, zero dagster_otel/opentelemetry imports, nothing at all:

git clone https://github.com/HirofumiTsuda/opentelemetry-instrumentation-dagster
cd opentelemetry-instrumentation-dagster
docker compose up -d

pip install opentelemetry-instrumentation-dagster
OTEL_SERVICE_NAME=quickstart \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
opentelemetry-instrument dagster asset materialize -f examples/definitions.py --select example_asset

Open http://localhost:16686 (Jaeger's UI), select the quickstart service, and there's the span -- example_asset, from a file that never imported this package or dagster-otel at all.

Teardown: docker compose down.

Usage

Same idea against your own pipeline: run your usual Dagster command through the opentelemetry-instrument launcher instead of running it directly.

OTEL_SERVICE_NAME=my_pipeline \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
opentelemetry-instrument dagster dev -f definitions.py

That's the entire setup. Every @op/@asset/@multi_asset/@asset_check/ @dbt_assets in definitions.py gets a span automatically, with no decorator, no import, no change to the file at all:

from dagster import asset, job, op

@op
def upstream_op(context) -> int:
    return 1

@op
def downstream_op(context, x: int) -> int:
    return x + 1

@asset
def my_asset(context) -> None:
    ...

@job
def my_job():
    downstream_op(upstream_op())

The launcher works the same way in front of dagster job execute, dagster-webserver, dagster-daemon, or any other Dagster entry point -- it's a drop-in prefix, not something specific to dagster dev.

What's covered

Configuration

Same standard OTel environment variables dagster-otel itself reads -- nothing this package adds on top:

See dagster-otel's own README for the full list and what each one actually does underneath -- this package doesn't wrap or reinterpret any of it, just triggers the same traced() that reads these itself.

multiprocess/k8s_job_executor

Each step in a multiprocess-executed run runs in its own, freshly spawned Python interpreter -- the opentelemetry-instrument launcher above already handles this correctly (verified against a real run, see docs/design.md), no extra setup needed.

k8s_job_executor is different: each step becomes a genuinely separate Kubernetes Pod, and the launcher's usual mechanism can't propagate into a brand new container the way it does into a spawned OS subprocess. This needs the instrumentation baked into the container image itself instead -- no opentelemetry-instrument prefix anywhere, because nothing in your own config controls the command k8s_job_executor constructs internally for each step Pod.

Using this with k8s_job_executor

In your own Dockerfile, after installing this package, copy opentelemetry-instrumentation's real sitecustomize.py into your venv's site-packages root:

RUN site_packages="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')" && \
    cp "$site_packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py" \
       "$site_packages/sitecustomize.py"

Python auto-imports any module literally named sitecustomize found directly in site-packages at interpreter startup -- so this alone is enough, in every container built from that image, including every step Pod k8s_job_executor launches from it. No PYTHONPATH, no .pth file, no launcher prefix anywhere in your run config or Dockerfile CMD.

(If your build process makes a sysconfig-based path awkward -- e.g. a venv at a path you already know ahead of time -- find /path/to/venv -maxdepth 4 -type d -name site-packages works just as well. Setting PYTHONPATH explicitly via k8s_job_executor's env_vars run config is an alternative to copying the file at all, if that fits your deployment better.)

dev/kubernetes/ is a complete, real-cluster-verified example of this exact pattern (Dockerfile, kind manifests, RBAC) -- written as this project's own verification harness, not a copy-paste-ready deployment template, but the Dockerfile's approach is the same one described above. See docs/design.md for the full reasoning and the negative-control verification.

Compatibility

Depends on dagster-otel and dagster >= 1.5, same floor as that project. Not independently version-matrix-tested beyond what dagster-otel itself covers -- if you hit an incompatibility, open an issue.

Why a separate package

dagster-otel's whole pitch is tracing without monkeypatching Dagster internals and without taking ownership of your op/asset definitions away from you. Auto-instrumentation is the opposite trade: zero code changes, in exchange for some framework patching and losing that per-function visibility. Both are legitimate, but they're different products for different people -- same split the OpenTelemetry Python ecosystem itself uses (opentelemetry-instrumentation-flask, -django, etc. are all separate packages from the manual API/SDK). See docs/design.md for the full reasoning.

Contributing

Issues and PRs welcome -- open an issue for bugs, missing coverage, or a backend that doesn't work as expected.

Roadmap

• @op/@asset/@multi_asset patched, verified against real Dagster execution across multiprocess and k8s_job_executor, both exported to a real Jaeger
• @asset_check patched (see What's covered)
• @dbt_assets covered for free -- it calls multi_asset internally
• opentelemetry_instrumentor entry point resolution verified
• Warn when dagster_dbt is already imported before instrument() runs
• Combined demo: dagster-prometheus-exporter metrics + zero-code traces, through Grafana Tempo (#24)
• Dispatch @dbt_assets calls to traced_dbt() for per-dbt-node span granularity (#14)
• Verify @dbt_assets end-to-end against a real dbt project (#6)
• Test that exceptions propagate correctly through the patched decorator (#11)
• Test the parameterized-decorator-without-name dispatch branch (#10)
• Test uninstrument() for @asset/@multi_asset, not just @op (#9)

License

MIT