silver-doodle

Lightning training runs with LightningCLI, run locally or on Kubeflow Trainer, with the config and metrics logged to MLflow.

Setup

uv sync --extra train  # train locally, and resolve configs locally before submitting
uv sync                # or: only submit to Kubeflow
cp .env.example .env   # set MLFLOW_TRACKING_URI and credentials

Run locally

uv run silver-doodle fit --config configs/example.yaml
uv run silver-doodle fit --config configs/example.yaml --trainer.max_epochs 2   # override anything
uv run silver-doodle fit --config configs/example.yaml --print_config           # show the full config

Run on Kubeflow

silver-doodle-submit takes the same arguments as silver-doodle:

uv run silver-doodle-submit --dry-run fit --config configs/example.yaml   # prepare only
uv run silver-doodle-submit fit --config configs/example.yaml                            # 1 GPU
uv run silver-doodle-submit fit --config configs/example.yaml --trainer.accelerator cpu  # CPU only
uv run silver-doodle-submit fit --config configs/example.yaml \
    --trainer.accelerator gpu --trainer.devices 2 --trainer.num_nodes 2

1. With the train extra, the full config is resolved locally with --print_config, so mistakes fail before submitting. Without it, the config files and overrides are sent as they are and resolved in the pod, and the pod size is read from the trainer settings in them.
2. The project you run from (the nearest pyproject.toml at or above the working directory) is built into a wheel with uv build, uncommitted changes included. That's this repo, or your own project using silver-doodle for its models and data: then the pod also installs the project's dependencies and the same silver-doodle version from PyPI. Outside any project (uvx in a plain folder), the pod only installs that silver-doodle version.
3. The pod installs the train dependencies and silver-doodle, then runs silver-doodle with the shipped config files.

The progress bar is turned off in the pod unless you set --trainer.enable_progress_bar. Pod resources come from the config: trainer.num_nodes nodes, each with trainer.devices GPUs when trainer.accelerator is auto (Lightning's default), gpu or cuda. A pod only sees the GPUs it requests, so auto requests them: 1 per node unless trainer.devices is a count. Use --trainer.accelerator cpu for a CPU-only job. If no GPU node is free, the job waits in Pending. The MLflow env vars are forwarded to the pod. Logs, the resolved config and the wheel are saved in logs/<timestamp>/.

MLFLOW_TRACKING_URI must point to a tracking server, not sqlite:///...: the URI ends up in the resolved config (it is MLFlowLogger's default), and a local database in the pod is lost with it.

Development

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check .