metaflow-prefect 0.1.0

Metaflow extension: deploy and run flows as Prefect deployments

metaflow-prefect

Deploy and run Metaflow flows as Prefect deployments.

metaflow-prefect generates a self-contained Prefect flow file from any Metaflow flow, letting you schedule, deploy, and monitor your pipelines through Prefect while keeping all your existing Metaflow code unchanged.

Install

pip install metaflow-prefect

Or from source:

git clone https://github.com/npow/metaflow-prefect.git
cd metaflow-prefect
pip install -e ".[dev]"

Quick start

python my_flow.py prefect create my_flow_prefect.py
python my_flow_prefect.py

Usage

Generate and run a Prefect flow

python my_flow.py prefect create my_flow_prefect.py
python my_flow_prefect.py

All graph shapes are supported

# Linear
class SimpleFlow(FlowSpec):
    @step
    def start(self):
        self.value = 42
        self.next(self.end)
    @step
    def end(self): pass

# Split/join (branch)
class BranchFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.branch_a, self.branch_b)
    ...

# Foreach fan-out
class ForeachFlow(FlowSpec):
    @step
    def start(self):
        self.items = [1, 2, 3]
        self.next(self.process, foreach="items")
    ...

Parametrised flows

Parameters defined with metaflow.Parameter are forwarded automatically:

python param_flow.py prefect create param_flow_prefect.py
python param_flow_prefect.py --message "hello" --count 5

How it works

metaflow-prefect generates a self-contained Prefect flow file from your Metaflow flow's DAG. Each Metaflow step becomes a @task. The generated file:

• runs each step as a subprocess via the standard metaflow step CLI
• passes --input-paths correctly for joins and foreach splits
• writes Metaflow artifacts to the Prefect UI as markdown artifacts with a ready-to-use retrieval snippet

Prefect UI: flow run timeline

The generated flow preserves the Metaflow DAG structure — foreach fan-outs appear as parallel task runs in the Prefect timeline:

Prefect UI: artifact retrieval snippets

After each step completes, a Prefect artifact is posted showing the Metaflow self.* artifact names and a one-liner to fetch each value:

Development

git clone https://github.com/npow/metaflow-prefect.git
cd metaflow-prefect
pip install -e ".[dev]"
pytest -v

License

Apache 2.0