metaflow-logs 0.1.0

Ships Metaflow task logs to a centralized backend via OpenTelemetry

metaflow-logs

A Metaflow extension that ships task stdout and stderr to any OpenTelemetry-compatible backend (Grafana Loki, Datadog, New Relic, Elasticsearch, Honeycomb, …). Each log line is tagged with full task context — flow name, run ID, step name, task ID, attempt — making logs filterable and correlatable from any OTel-aware tool.

Quick start

pip install metaflow-logs

Add the decorator to any step:

from metaflow import FlowSpec, step
from metaflow_extensions.logs.plugins.decorator import LogsDecorator

class MyFlow(FlowSpec):
    @LogsDecorator()
    @step
    def train(self):
        print("epoch 1 complete")
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

Point it at your OTel collector:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
python flow.py run

Each log line arrives in your backend tagged with:

Attribute	Example
metaflow.flow	MyFlow
metaflow.run_id	123
metaflow.step	train
metaflow.task_id	456
metaflow.attempt	0
metaflow.log_source	task or runtime

Configuration

All configuration is via standard OTel environment variables:

Variable	Default	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318	OTel collector endpoint
OTEL_EXPORTER_OTLP_HEADERS	—	Auth headers (e.g. Authorization=Bearer ...)
OTEL_SERVICE_NAME	metaflow	Service name in your backend

Falls back to a console exporter if opentelemetry-exporter-otlp-proto-http is not installed.

How it works

Reads Metaflow's structured MFLOG files (MFLOG_STDOUT, MFLOG_STDERR) at step completion and emits each line as an OTel LogRecord. Runtime logs (Metaflow internals) are emitted at DEBUG severity; task logs (your code's stdout/stderr) at INFO.

Development

git clone https://github.com/npow/metaflow-logs
cd metaflow-logs
pip install -e .
pytest tests/ -v