cauldron-ml 0.6.2

A CLI tool for building and deploying Kubeflow Pipelines in GCP Vertex AI

CauldronML: A CLI tool for building and deploying Kubeflow Pipelines in GCP Vertex AI

Quickstart (10 mins)

Before you start, we recommend installing a python environment manager. Cauldron uses uv in the build process for it's speed, but you don't need to use it locally. We prefer pyenv for it's user-friendliness and automatic switching between projects.

If you don't already have one installed, you can install pyenv in one line.

Installation requirements

Cauldron works with unix-based systems like MacOS and Linux. It does not currently work on Windows. For windows users, we recommend installing Windows Subsystem for Linux (WSL) but we have not tested this.

Install pyenv

curl https://pyenv.run | bash

Install a compatible python version and virtual envioronment

Python >=3.10 is supported, although python >=3.12 is recommended as this will allow you to use the latest versions of kubeflow (kfp) and google-cloud-aiplatform.

Start with a blank folder

mkdir my-project
cd my-project

Install python 3.12, create a new virtualenv and set it to auto-activate in the current folder

pyenv install 3.12
pyenv virtualenv 3.12 my-project
pyenv local my-project

Install cauldron-ml

pip install git+https://github.com/dprice80/cauldron-ml.git

Verify the installation and check the CLI tool is registered

caul --help

Help output should be displayed

Initialise Cauldron

caul init

This will create a yaml file called ~/.caulprofile. This file should be modified by the user with the following settings:

docker-repo: <repo-address>/subfolder  # no trailing slash
docker-base-image: python:3.12-slim  # By default, cauldron pipelines work with slim versions of python images (tested with 3.12).
user-prefix: jbloggs
production-project-name: foo-bar-prod
sandbox-project-name: foo-bar-sandbox
production-service-account: my-sa@[e-mail]
sandbox-service-account: my-sa@[e-mail]

Note that docker-repo should include the namespace where the images are stored. The production-project-name and production-service-account can be the same as the sandbox accounts. In that case prod pipelines would run alongside dev pipelines in the same GCP project. However, this setup is not recommended. It is much safer to run a separate prod project, and for the deployment to prod to be executed by an automated workflow such as github actions. We will add examples to this effect shortly.

Now you are ready to build your first project. Cauldron has a built-in templating system, with one working example. More examples will be added at a later date.

caul create

This will give you a working example template. You should try building, deploying and running this template without modification to ensure installation succeeded and all permissions are set correctly on your account. The template will run with your user-prefix (from ~/.caulprofile).

CauldronML uses a two stage docker build to save time when making changes to your pipeline. First, build the base image.

caul build --base

Then build the pipeline image.

caul build

NOTE: Running caul build with no base image present in the local or remote docker repo will trigger the automatic build of the base image.

Upload the images to your docker repo.

caul deploy

Run the pipeline.

caul run

You should now have a running pipeline.

CI/CD Setting up GitHub Actions Workflows

Our intention is for users to build their own github actions that trigger when code is pushed to main via a PR. CauldronML works with kaniko to build both the base image and pipeline image. We will add detailed instructions on how to do this shortly.