silver-doodle

Lightning training runs with LightningCLI, run locally or on Kubeflow Trainer, with the config and metrics logged to MLflow.

Setup

uv sync --extra train  # train locally, and resolve configs locally before submitting
uv sync                # or: only submit to Kubeflow
cp .env.example .env   # set MLFLOW_TRACKING_URI and credentials

Run locally

uv run silver-doodle fit --config configs/example.yaml
uv run silver-doodle fit --config configs/example.yaml --trainer.max_epochs 2   # override anything
uv run silver-doodle fit --config configs/example.yaml --print_config           # show the full config

Run on Kubeflow

silver-doodle-submit takes the same arguments as silver-doodle:

uv run silver-doodle-submit --dry-run fit --config configs/example.yaml   # prepare only
uv run silver-doodle-submit fit --config configs/example.yaml --trainer.enable_progress_bar false
uv run silver-doodle-submit fit --config configs/example.yaml \
    --trainer.accelerator gpu --trainer.devices 2 --trainer.num_nodes 2

1. With the train extra, the full config is resolved locally with --print_config, so mistakes fail before submitting. Without it, the config files and overrides are sent as they are and resolved in the pod, and the pod size is read from the trainer settings in them.
2. The local code, uncommitted changes included, is built into a wheel with uv build.
3. The pod installs the train dependencies and the wheel, then runs silver-doodle with the shipped config files.

Pod resources come from the config: trainer.num_nodes nodes, each with trainer.devices GPUs when trainer.accelerator is gpu/cuda. The MLflow env vars are forwarded to the pod. Logs, the resolved config and the wheel are saved in logs/<timestamp>/.

MLFLOW_TRACKING_URI must point to a tracking server, not sqlite:///...: the URI ends up in the resolved config (it is MLFlowLogger's default), and a local database in the pod is lost with it.

Development

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check .