Pipewright

Configurable, governed MLOps on Google Cloud. Build, release, run and monitor models from configuration, with a human in the loop for every retrain decision.

Pipewright is open source under the Apache License 2.0 and maintained by Fluxor Software.

Status: 1.0.0. Working end to end on Google Cloud, including private environments with customer-managed keys, enforced organization policies and an enforced VPC Service Controls perimeter. The configuration format is stable (STABILITY.md). Releases are on GitHub (the wheel as a release file, the launcher image in GitHub Container Registry) and on PyPI as pipewright-mlops.

At a glance

_{Editable source: docs/images/pipewright-architecture.drawio (draw.io, official Google Cloud icons).}

The same environment on private GKE. One setting in the manifest (orchestrator: kfp) runs the same pipelines on Kubeflow Pipelines in a private GKE cluster, with Ray on KubeRay and online serving on KServe. Templates, contracts, the CLI and everything outside the cluster stay the same.

_{Editable source: docs/images/pipewright-gke-architecture.drawio.}

The platform team builds the environment once and publishes an environment manifest: the names of everything a team may use. ML teams then work only with their own configuration and code; the pipewright CLI and the shared launcher do the rest.

Building blocks: what you need on Google Cloud

Pipewright does not create infrastructure. Before an ML team runs pipewright, the platform side provides these resources in each environment's project and describes them in the manifest (src/pipewright/config/schemas/manifest.schema.json).

APIs to enable: aiplatform, artifactregistry, bigquery, cloudscheduler, iam, iamcredentials, logging, monitoring, pubsub, run, storage (all .googleapis.com), plus cloudkms for customer-managed keys and container for private GKE.

Setting up the platform side. This repository does not include Terraform for the environment. If you would like help building one, or the Terraform we use, email hello@fluxorsoft.com or open a GitHub Discussion: we are happy to share, and it helps us shape what comes next.

The problem

Most organisations that put machine learning into production on Google Cloud hit the same gaps, team after team:

1. Every ML team rebuilds the same plumbing. Triggers, release scripts, registry conventions, drift monitoring, alerting and IAM are written again for each use case. Months pass between a working notebook and a model in production, and every team does it slightly differently.
2. A pipeline run is not a release. Ask "what exactly is running in production, and can we go back?" and the answer is usually a guess: images tagged latest, templates overwritten in place, no single record of what was approved.
3. Platform and ML teams block each other. ML teams wait on infrastructure tickets for buckets, datasets and permissions; platform changes break pipelines nobody told them about. There is no contract between the two.
4. Settings live in code. A gate threshold, a drift limit or a machine size needs a code change and an engineer. Dev and production quietly drift apart, so the model you tested is not quite the model you ship.
5. Drift alerts lead to blind retraining, or to nothing. Drift says the inputs changed, not why. An upstream data bug, a seasonal blip and a real change look the same. Automatic retraining bakes the bug into the next model or fits the blip; retraining on the same data does not improve anything. Ignoring alerts is worse.
6. Governance is assembled after the fact. Auditors and model-risk teams ask which data trained a model, who approved it, where it runs, which key encrypts it and why it was retrained. The answers come from spreadsheets and memory.
7. Regulated environments break the usual tools. Customer-managed keys on every query, no public endpoints, VPC Service Controls and enforced organization policies stop most off-the-shelf MLOps tooling at the door.

How Pipewright closes them

What it looks like

A use case's operations file, owned and reviewed by the ML team:

# usecases/churn/config/operations.yaml
monitoring:
  output_table: churn_model_monitoring
  drift: { enabled: true, method: psi, threshold: 0.2 }
  performance: { enabled: true, metric: auc, min: 0.7,
                 labels: { table: churn_outcomes, key: customer_id, label: churned } }
  alerts: { notify: [ml-team@example.com], events: [monitoring.alert, training.rejected] }

retraining:
  mode: propose                     # off | propose | candidate (trains, never promotes)
  when: [drift, performance]
  decisions: { notify: [ml-team@example.com], publish: true }
  approvals: { required: [ml-leads, model-risk] }
  snooze: { max_days: 14 }

And the lifecycle, from the MLOps repository:

pipewright new-usecase --name churn --tenant retention --owner team-retention
pipewright validate     --env dev                     # schemas, references, cross-checks, offline
pipewright doctor       --env dev                     # read-only checks against the live project
pipewright tables sync  --env dev                     # create tables; additive changes only
pipewright release      --env dev                     # build, pin by digest, verify, go live
pipewright run          --env dev --usecase churn --pipeline inference --source-uri gs://...
pipewright monitor status --env dev --usecase churn   # drift, performance, open retrain cases
pipewright retrain show   --env dev --usecase churn   # the decision pack
pipewright retrain approve --env dev --usecase churn --reason "real shift after the price change"

Features

How it works

Two personas, one contract between them (the diagram at the top shows the flow):

• The platform team owns durable, shared and security-relevant resources, and publishes the environment manifest.
• ML teams own pipeline code, configuration, tables inside their dataset, releases, schedules and model versions. Neither side edits the other's files.
• The launcher is deployed once per environment and never redeployed for a model release: a release only moves a pointer to a new, verified contract.
• Every step runs as the team's own identity, with the environment's key and network applied from the manifest.

CONTEXT.md describes the design, the decisions behind it and every interface in detail.

Your code and Pipewright

Teams write their pipelines and ML code their own way. Pipewright needs only a small contract at the edges:

Pipewright checks the gates itself. Your evaluation step may apply the gates, but Pipewright does not take its word for it. Before a version becomes the champion (pipewright model approve) or is served (pipewright deploy-model), Pipewright recomputes every metric gate, slice gate and the champion margin from the version's evaluation record, against the configuration in force now. A rewritten evaluation step that drops a gate cannot promote a failing model, and pipewright model list shows each version's result. The record:

{
  "metrics":  {"auc": 0.81, "calibration_error": 0.03},
  "slices":   {"contract_length_months": {"12": {"rows": 400, "auc": 0.79}}},
  "champion": {"auc": 0.80, "_version": "3"},
  "decision": "approve"
}

slices is needed for slice gates; champion holds the champion's metrics on the same test data, or null when there was no champion yet. The churn example writes this record in its evaluate and register steps.

Ray steps

A step that needs a cluster asks for one in configuration and uses it in code. The cluster exists only for that step of that run:

steps:
  tune-on-ray:
    ray: { head_machine: n1-standard-4, workers: 1 }   # within the platform's limits; may differ per environment
components:
  ray_requirements: ray-requirements.txt               # extra packages for the cluster's nodes

from pipewright_runtime import ray_cluster, ray_job

with ray_cluster(ml_config, "tune-on-ray", run_id) as cluster:      # created by the launcher, deleted after
    ray_job(ml_config, cluster, f"python -m ray_tune_job {trials.uri}", run_id=run_id)

• Who makes the cluster: the launcher, as the tenant's runner identity, from the live release (the step only names itself). It applies the environment's key and network, and labels the cluster with the run.

• What it runs: a Ray image pipewright release builds for the use case, on the platform's Ray on Agent Platform base image, with the component code and ray_requirements.

• Cleanup: the step deletes the cluster when it ends, even on errors; the launcher's tick deletes any cluster whose run ended or whose max_minutes passed, so a killed step never leaves one running.

• The platform's side: an optional ray block in the manifest (environment and per tenant): whether Ray is on, the Ray version, base image, a Private Service Connect network attachment, and limits on machines, workers, accelerators and minutes. Without it: a cluster per run on a public endpoint, the runner identity, a prebuilt image, small limits.

• What the platform must grant (not in the starter Terraform):

  • the launcher's identity: aiplatform.persistentResources.create, delete, get and list (a custom role; roles/aiplatform.user only gets and lists);
  • each tenant runner, on itself: roles/iam.serviceAccountUser (it reaches its own cluster's job API only if it may act as itself) and roles/iam.serviceAccountOpenIdTokenCreator (the step signs the ID token it asks the launcher with: Agent Platform workers' metadata server does not reliably issue one).

  Pipewright Enterprise's Terraform grants all of it, with the Private Service Connect network for private environments.

• A Ray head needs at least 16 vCPUs and more than 18 GB of memory: with fewer vCPUs the cluster runs but its job API never answers (found live), and Agent Platform refuses less memory. validate checks both.

• A Ray step does not size itself: no cpu, memory or gpu on a Ray step (Agent Platform runs sized steps where the metadata server issues no ID tokens, so the step could not ask the launcher); size the cluster instead.

• Checks: validate refuses a step over the limits and a Ray client that does not match the clusters (ray[default] pinned in the component requirements, with google-cloud-aiplatform[ray]).

• On private GKE (orchestrator: kfp): the same ray_cluster() and ray_job() calls get a KubeRay RayCluster on the cluster's Ray node pool, from the mirrored Ray image, and the job is submitted inside the cluster. The platform can instead give a team a standing cluster (tenants.<t>.ray.mode: standing in the manifest's ray block): ray-<tenant>, kept by the platform, whose autoscaler adds workers for a job and removes them after the idle minutes. A release records the mode, so changing it makes a new release. The step code is the same.

pipewright new-usecase --from ray starts from an example that runs the churn tuning trials in parallel on Ray.

Development and pipelines: while developing, data scientists use a standalone Ray cluster of their own (made in the console, used from a notebook with ray.init("vertex_ray://<cluster>"), Python and Ray versions matching the cluster). Pipelines never use it: every run, dev to prod, gets its own cluster from Pipewright. The same Ray code runs in both; only who owns the cluster changes.

Features from feature tables

Instead of one prepared table per use case, a use case can name the features it needs and the BigQuery feature tables that hold them. Feature tables belong to whoever produces the features (a data team, or the team itself); Pipewright only reads them. The recommended way to use features, how a feature table must look and the mistakes to avoid: docs/FEATURES.md.

# config/data.yaml
data:
  contract: { ... }                       # unchanged: every column the model and the checks see
  features_from:
    entity: { columns: [customer_id], time_column: snapshot_date }    # in the run's source, the entity table
    sources:
      - table: bq://features.customer_profile          # entity columns, feature_timestamp, one column per feature
        features: [tenure_months, monthly_charges]
      - table: bq://features.support_activity
        features: [num_support_tickets]

• The run's source is the entity table: who, as of when, and the label (pipewright run --source-uri bq://...). Each row gets the newest value of each feature dated at or before its cut-off, so nothing from after it reaches training. A date cut-off is the start of that day (UTC).
• Scoring uses the same query, so training and batch scoring read features the same way. A scoring table usually carries the cut-off column as the scoring date, which also makes the scores reproducible. Without it (the contract must then mark that column required: false) the cut-off is the query time.
• One query, in BigQuery, run as the runner with the environment's key; everything after ingest (the data contract, validation, training, gates, monitoring) is unchanged. A data.features.resolved event records the tables, features, query job and time, and how many rows found a value.
• Checks: validate checks the block against the data contract (every feature in the contract, no clashes, not the label, a date or timestamp cut-off) and refuses landing-file triggers, since the source must be a bq:// table; doctor checks each feature table has the columns, with types the contract accepts.
• The platform decides where features may come from (manifest): tenants.<t>.features.datasets lists a tenant's feature datasets, and features.required: true makes feature tables the only source of model features. validate refuses anything else.

Why tables and not a feature store product: the offline store of Google's Agent Platform Feature Store is BigQuery itself, and its training-data fetch is still in Preview. Registered feature groups are a second kind of source that resolves to the same tables: - feature_group: customer_profile with registered feature names, read from the Feature Registry at each run. pipewright new-usecase --from features starts from an example with two feature tables.

Synthetic data

Off until your team needs it. A synthetic section in the use case switches it on; without it nothing is built, compiled or released for it, and the use case behaves exactly as before.

# config/synthetic.yaml
synthetic:
  enabled: true                          # may differ per environment
  purpose: test-data                     # sets the gates: test-data stays in the organisation, sharing leaves it
  source_uri: bq://${project}.${resources.bigquery_dataset}.customers    # read like training data, point in time
  output_table: synthetic_customers      # in outputs.bigquery.tables: the contract's columns plus run_id
  columns:
    customer_id: { rule: key, prefix: "SYN-" }     # keys and personal data are never learned

• Two tiers. learned: an engine learns from the real data inside the environment, in a synthetic pipeline Pipewright provides (no code to write): pipewright run --pipeline synthetic, or a schedule with pipeline: synthetic. contract: no real data at all; pipewright synthetic generate --out rows.csv writes rows from the data contract and the column rules, for development, CI and demos.
• Engines. MOSTLY AI's Synthetic Data SDK in local mode (Apache-2.0) is the default; it makes no network calls. SDV is available as an engine your team installs itself (synthetic.requirements), under its own licence (Business Source Licence 1.1); Pipewright never ships it.
• Gated, never trusted. Each candidate (by default the engine at sampling temperatures 1.0 and 1.3, in parallel) is scored the same way: fidelity, utility (a model trained on synthetic data tested on a real holdout) and privacy (exact copies, distance to the closest record, singling-out, linkability and inference attacks). Pipewright recomputes the purpose's gates and publishes the best candidate that passes all of them, or nothing. pipewright synthetic gates lists them; privacy gates may only be made stricter.
• Never learned: personal-data columns (data_protection.personal_data_columns) and keys get new values from rules; a real holdout is kept back from the engine; the generator stays inside the step.
• Released like any pipeline: the synthetic pipeline runs on two images Pipewright builds (the engine, and the evaluation), recorded in the release contract by digest, verified, and copied by promote.
• Relational and sequential (mode: relational): the entity table and its child tables in one model, the feature tables of data.features_from and event tables in synthetic.sequences (ordered per entity). Keys stay consistent across tables, history keeps its timing against each cut-off, and the gates are measured on the point-in-time join of the synthetic tables; each table goes to its own output table.
• Events: synthetic.evaluated, synthetic.rejected and synthetic.generated carry the gates, the chosen candidate and where the rows went.

Approval of a synthetic release by named approver groups, releasing it into other environments, evidence and per-environment privacy policy are in Pipewright Enterprise.

Promotion between environments

What runs in production is what was tested: pipewright promote --from test --to prod --usecase <name> takes the release live in test (and its approved model) to production without rebuilding anything.

• Allowed source: only from the environment the destination names (promotion: { from: test } in its environment file), and from a clean checkout of the source release's commit.
• Destination policies first: the use case is resolved under production's manifest (approval policies, feature rules, Ray limits); a refusal stops before anything is copied.
• The same code: images are copied by digest and checked. Templates are compiled for production and must equal the tested ones with only the registry address changed; an override that changes a pipeline's shape is refused unless accepted with --accept-template-change (recorded).
• The same model (model.promotion: model): the approved version's files, verified, registered in production as a candidate; pipewright model approve there makes it the default. A model's first version needs --approve with the promotion, since Model Registry makes it the default. With model.promotion: retrain, production trains its own candidate.
• Safe to repeat, recorded, reversible: the contract is written once and the pointer moved last; repeating is a no-op; promoted_from and a release.promoted event record what came from where, by whom; pipewright rollback goes back without rebuilding.

Getting started

What you need

• On your machine: Python 3.11, uv, Docker (component images), and the Google Cloud CLI or Application Default Credentials.
• On Google Cloud: an environment with the building blocks above and its environment manifest. docs/ONBOARDING.md lists what the platform team hands to each ML team.

Install a release

pip install "pipewright-mlops[gcp]==1.0.0"     # the command is `pipewright`

Each release is also on GitHub, with the wheel and sdist as release files (gh release download v1.0.0 -R fluxorsoft/pipewright). Pin the version everywhere, CI included, directly from PyPI or through your organisation's package index. The launcher image for platform teams is in GitHub Container Registry (ghcr.io/fluxorsoft/pipewright-launcher), with its digest in the release notes.

Install from source

git clone https://github.com/fluxorsoft/pipewright.git
cd pipewright
uv sync --all-extras                # Python 3.11 virtual environment with the gcp and launcher extras
uv run pipewright --help

Or install the CLI into another environment from the checkout: pip install "<path to the checkout>[gcp]".

Your first use case

From an MLOps repository with an environments/<env>.yaml that points at your manifest:

apiVersion: pipewright/v1
kind: Environment
metadata: { name: dev }
spec:
  manifest: gs://<artifacts-bucket>/pipewright/environment.json

pipewright new-usecase --name churn --tenant <your-tenant> --owner <your-team>   # copies the churn example
pipewright validate --env dev
pipewright doctor   --env dev
pipewright tables sync --env dev
git commit -am "churn: first release"     # releases are named after the commit
pipewright release  --env dev

The churn example (examples/churn/) is a complete reference: training and inference pipelines, components, a data contract, gates, monitoring, retrain decisions, table schemas, tests and generated sample data. The full walkthrough, including what the platform team hands over, is in docs/ONBOARDING.md.

Help with the YAML

Credentials: Application Default Credentials, or GCLOUD_CONFIG=<gcloud configuration> for refreshable gcloud user credentials. Behind a TLS-inspecting proxy, set REQUESTS_CA_BUNDLE, GRPC_DEFAULT_SSL_ROOTS_FILE_PATH and PIPEWRIGHT_BUILD_CA_BUNDLE.

Commands

Every command takes --env and most take --usecase. pipewright <command> --help lists the options.

Open source and Enterprise

Pipewright is open core. Everything an ML team uses is open source; Pipewright Enterprise adds what regulated organisations need around it, through published extension points (manifest fields, launcher routes, CLI plugins), never by patching this package.

Enterprise enquiries: fluxorsoft.com/pipewright.

Project status and roadmap

Google Cloud only, by design. Pipelines are Kubeflow Pipelines v2; bring any Python ML framework (the example uses PyTorch).

Development

uv sync --all-extras
uv run pytest                     # unit tests
uv run ruff check .               # lint
uv run --no-project --with-requirements examples/churn/tests/requirements.txt python -m pytest examples/churn/tests

CONTEXT.md is the place to start before changing anything: requirements, decisions, design and conventions, with stable IDs to refer to. See CONTRIBUTING.md for how to propose changes, and SECURITY.md to report a vulnerability.

License

Copyright 2026 Fluxor Software Limited. Licensed under the Apache License, Version 2.0; see NOTICE. "Pipewright" and "Fluxor" are names of Fluxor Software Limited; the license does not grant permission to use them beyond describing the origin of the work.