Skip to main content

Apache Airflow operator for running Google Cloud Run Jobs using green energy

Project description

logo
VertFlow

Run Docker containers on Airflow using green energy

Video Demo

📖 About

VertFlow is an Airflow operator for running Cloud Run Jobs on Google Cloud Platform in green data centres.
Cloud Run is a serverless container runtime, meaning you BYO Docker image and emit carbon only when the job is running. This is easier, cheaper and greener than managing a Kubernetes cluster spinning 24/7.

Not all data centres are created equal.
Data centres run on electricity generated from various sources, including fossil fuels which emit harmful carbon emissions. Some data centres are greener than others, using electricity from renewable sources such as wind and hydro.
When you deploy a container on Airflow using the VertFlow operator, it will run your container in the greenest GCP data centre possible.

ℹ️ Use VertFlow on Cloud Composer 2 to save even more money and CO2.

🔧 How to install

  1. pip install VertFlow on your Airflow instance.
  2. Ensure your Airflow scheduler has outbound access to the public internet and the roles/run.developer Cloud IAM role.
  3. Get an API Key for CO2 Signal, free for non-commercial use. Store in an Airflow variable called VERTFLOW_API_KEY.

ℹ️ If you're using Cloud Composer, these instructions may be helpful:

🖱 How to use

Use the VertFlowOperator to instantiate a task in your DAG. Provide:

  • The address of the Docker image to run.
  • A runtime specification, e.g. timeout and memory limits.
  • A set of allowed regions to run the job in, based on latency, data governance and other considerations. VertFlow picks the greenest one.
from VertFlow.operator import VertFlowOperator
from airflow import DAG

with DAG(
        dag_id="hourly_dag_in_green_region",
        schedule_interval="@hourly"
) as dag:
    task = VertFlowOperator(
        image_address="us-docker.pkg.dev/cloudrun/container/job:latest",
        name="hello-world",
        allowed_regions=["europe-west1", "europe-west4"],
        command="echo",
        arguments=["Hello World"],
        service_account_email_address="my-service-account@embroidered-elephant-739.iam.gserviceaccount.com",
        ...
    )

🔌🗺 Shout out to CO2 Signal

VertFlow works thanks to real-time global carbon intensity data, gifted to the world for non-commercial use by CO2 Signal.

🤝 How to contribute

Found a bug or fancy resolving an issue? We welcome Pull Requests!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

VertFlow-0.1.4.tar.gz (13.9 kB view details)

Uploaded Source

Built Distribution

VertFlow-0.1.4-py3-none-any.whl (17.5 kB view details)

Uploaded Python 3

File details

Details for the file VertFlow-0.1.4.tar.gz.

File metadata

  • Download URL: VertFlow-0.1.4.tar.gz
  • Upload date:
  • Size: 13.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.6

File hashes

Hashes for VertFlow-0.1.4.tar.gz
Algorithm Hash digest
SHA256 2cbe0f80e22a813d38ae76d8377b81e3373a670f745a541e526b7c09dccd645b
MD5 b9936920d11a286aba40a548784b362a
BLAKE2b-256 58011b937a0c0c269113777b3f4a464fa1333e11be6e7f4469abea848237f4fd

See more details on using hashes here.

Provenance

File details

Details for the file VertFlow-0.1.4-py3-none-any.whl.

File metadata

  • Download URL: VertFlow-0.1.4-py3-none-any.whl
  • Upload date:
  • Size: 17.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.6

File hashes

Hashes for VertFlow-0.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 5d293663dd35aedf01f0aafd694716bc4cb39014fc4abefa61022265cb4916d0
MD5 de14b1caff68a342662656a69aedc938
BLAKE2b-256 77fb83801dcff00b36e1e67900fe8e742524d86129d88da64633c2f6c8b3b2d5

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page