silver-doodle

Lightning training runs with LightningCLI, run locally or on Kubeflow Trainer, with the config and metrics logged to MLflow.

Setup

uv sync --extra kubeflow
cp .env.example .env   # set MLFLOW_TRACKING_URI and credentials

Run locally

uv run silver-doodle fit --config configs/example.yaml
uv run silver-doodle fit --config configs/example.yaml --trainer.max_epochs 2   # override anything
uv run silver-doodle fit --config configs/example.yaml --print_config           # show the full config

Run on Kubeflow

silver-doodle-submit takes the same arguments as silver-doodle:

uv run silver-doodle-submit --dry-run fit --config configs/example.yaml   # prepare only
uv run silver-doodle-submit fit --config configs/example.yaml --trainer.enable_progress_bar false
uv run silver-doodle-submit fit --config configs/example.yaml \
    --trainer.accelerator gpu --trainer.devices 2 --trainer.num_nodes 2

1. The full config is resolved locally with --print_config, so mistakes fail before submitting.
2. The local code, uncommitted changes included, is built into a wheel with uv build.
3. The pod installs the dependencies and the wheel, then runs silver-doodle <subcommand> --config <resolved config>.

Pod resources come from the config: trainer.num_nodes nodes, each with trainer.devices GPUs when trainer.accelerator is gpu/cuda. The MLflow env vars are forwarded to the pod. Logs, the resolved config and the wheel are saved in logs/<timestamp>/.

MLFLOW_TRACKING_URI must point to a tracking server, not sqlite:///...: the URI ends up in the resolved config (it is MLFlowLogger's default), and a local database in the pod is lost with it.

Development

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check .