metaflow-logs

A Metaflow extension that ships task stdout and stderr to any OpenTelemetry-compatible backend (Grafana Loki, Datadog, New Relic, Elasticsearch, Honeycomb, …). Each log line is tagged with full task context — flow name, run ID, step name, task ID, attempt — making logs filterable and correlatable from any OTel-aware tool.

Quick start

pip install metaflow-logs

Add the decorator to any step:

from metaflow import FlowSpec, step
from metaflow_extensions.logs.plugins.decorator import LogsDecorator

class MyFlow(FlowSpec):
    @LogsDecorator()
    @step
    def train(self):
        print("epoch 1 complete")
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

Point it at your OTel collector:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
python flow.py run

Each log line arrives in your backend tagged with:

Configuration

All configuration is via standard OTel environment variables:

Falls back to a console exporter if opentelemetry-exporter-otlp-proto-http is not installed.

How it works

Reads Metaflow's structured MFLOG files (MFLOG_STDOUT, MFLOG_STDERR) at step completion and emits each line as an OTel LogRecord. Runtime logs (Metaflow internals) are emitted at DEBUG severity; task logs (your code's stdout/stderr) at INFO.

Development

git clone https://github.com/npow/metaflow-logs
cd metaflow-logs
pip install -e .
pytest tests/ -v