min-obproject 0.1.0

Reusable Metaflow flow templates and configurations

min-obproject

This obproject demonstrates

1. Configuration patterns for machine learning workflows where there are both many parameters inducing a need for a config file, and you want to dynamically override config values.
2. How to build Metaflow flow templates.

Repository Structure

min-obproject/
├── src/                          # Shared Python modules
│   └── flow_templates.py         # Reusable flow base classes
├── flows/                        # Individual Metaflow workflows
│   ├── 01-config-override/       # Example: basic config override pattern
│   └── 02-projectflow-inheritance/  # Example: template inheritance
└── obproject.toml                # Outerbounds project configuration

Key Features

• Makes use of Metaflow Configs.
• ConfigurableNeuralNetworkFlow: Base class example that auto-resolves a config.json assumed to be in an inheriting flow's directory. This pattern helps downstream flows leverage configs for automating experiments with many parameters, while avoiding a lot of boilerplate in FlowSpec-inheriting classes.
• Dynamic Config Resolution: Configs can be selectively overridden using individual Metaflow Parameters. This way, we get the best of both worlds, and individual parameters of the config files can be overridden with arbitrarily complex logic that library authors write once, and downstream flow consumers can operate.

Development Setup

Install as Editable Package (Local Development)

# Install uv if needed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install package in editable mode
uv pip install -e .

# With dev dependencies
uv pip install -e ".[dev]"

Now src modules are importable from anywhere:

from src.flow_templates import NeuralNetworkFlow

Run Flows

Always run from project root:

python flows/02-projectflow-inheritance/flow.py run
python flows/02-projectflow-inheritance/flow.py run --with kubernetes

Publishing to PyPI

Build and Publish

# Build distribution
uv build

# Publish to PyPI
uv publish

# Or publish to test PyPI first
uv publish --index-url https://test.pypi.org/legacy/

Use Published Package in Production

from metaflow import FlowSpec, step, pypi_base

@pypi_base(packages={"min-obproject": "0.1.0"})
class MyFlow(FlowSpec):
    ...

Example Usage

from metaflow import step
from obproject import ProjectFlow
from src.flow_templates import ConfigurableNeuralNetworkFlow

class MyTrainingFlow(ProjectFlow, ConfigurableNeuralNetworkFlow):
    
    @step
    def start(self):
        self._resolve_config()  # Loads config.json from flow directory
        print(f"Training with lr={self.config['lr']}")
        self.next(self.end)
    
    @step
    def end(self):
        pass

Create config.json in the same directory as your flow:

{
    "train_args": {
        "lr": 0.001,
        "optimizer": "adam"
    }
}

Override config values via CLI:

python flow.py run --lr 0.01