Skip to main content

Google Cloud Workflows wrapper

Project description

Goblet Workflows (Beta)

PyPI PyPI - Python Version

Goblet Workflows is a wrapper around GCP Workflows, which is a fully-managed orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.

While GCP Workflows is powerful, it requires learning a complex yaml DSL, which makes it hard to manage and scale, and has a steap learning curve to get started.

Goblet Workflows uses similar syntax that you can expect from airflow and prefect, to make it easy to write, deploy, and manager your workflows, while leveraging the severless nature of GCP workflows.

See below for a simple workflow example.

from goblet_workflows.workflow import Workflow
from goblet_workflows.steps import AssignStep, BQStep

w = Workflow("bigquery-step-example", params=["column"])

upack_args = AssignStep(
    w,
    "upack_args",
    column="${column}"
)

bq_step = BQStep(w, "bq_step", "select {column} from DATASET limit 1")

bq_step_2 = BQStep(w, "bq_step_2", "select colomn2 from DATASET2 where column1 = {bq_step[0]} ")

upack_args > bq_step > bq_step_2

Not to deploy you simply have to run the command goblet_workflows deploy, which will yield the workflow below.



Getting started

To install run:

pip install goblet_workflows

To deploy a workflow create a main.py file where you define your workflow and steps.

Then run

goblet_workflows deploy

To view the yaml run

goblet_workflows print

Examples

Goblet Workflow Examples

Features

  • Generic Steps
  • Branches
  • Deployment of Workflows and Schedules
  • GCP Connectors

GCP Workflows

Overview

Syntax Cheat Sheet

Issues

Please file any issues, bugs or feature requests as an issue on our GitHub page.

Roadmap

☑ Http Steps
☑ GCP Generic Connectors
☑ BQ Connector
☑ Dataform Connector
☑ Deploy
☑ Deploy Arbitrary File
☐ Execute
☐ Delete
☑ Schedule
☐ Conditions
☐ Subworkflows
☐ Parameter typing

Want to Contribute

If you would like to contribute to the library (e.g. by improving the documentation, solving a bug or adding a cool new feature) submit a pull request.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

goblet_workflows-0.1.1.tar.gz (15.0 kB view details)

Uploaded Source

Built Distribution

goblet_workflows-0.1.1-py3-none-any.whl (14.7 kB view details)

Uploaded Python 3

File details

Details for the file goblet_workflows-0.1.1.tar.gz.

File metadata

  • Download URL: goblet_workflows-0.1.1.tar.gz
  • Upload date:
  • Size: 15.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.4

File hashes

Hashes for goblet_workflows-0.1.1.tar.gz
Algorithm Hash digest
SHA256 04852137e02cbfe419e6a5273459a2525635ca97dcb364f40b3594f5b32ebb70
MD5 478ba5e5fa13b03439e711e8678a657d
BLAKE2b-256 fd6554359ea5c42d31f787a4d8a7d24e3f9fa00dd2d8be9b199cbffb2882584e

See more details on using hashes here.

File details

Details for the file goblet_workflows-0.1.1-py3-none-any.whl.

File metadata

File hashes

Hashes for goblet_workflows-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 6f0e3548070bc62a2bc09dcd152b39f22bf2232f42e545239e683b0ff0dcb4ee
MD5 12448f5625ca4e3ae49d944d6e8fcfdf
BLAKE2b-256 8716e47eb333a6eda83b7defa3eeac0ec1b8e2463bbed63682d5c6960ec18543

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page