Skip to main content

Fuga - Helper toolset for Google Cloud Cimposer

Project description

(This project is in public-beta status)

Fuga ['fu:ga]

pypi CircleCI Documentation Status

Fuga is a toolset (and API wrappers) for Google Cloud Composer (Airflow), which allows Composer users to develop/deploy workflows easier and in more organized way.


Install Requirements

Note: pyenv always provides an entrypoint for both python2/3 and throw an error when one doesn't actually exists, which makes gcloud cli crashes at its runtime. To prevent this, use something like pyenv global 3.x.x 2.x.x to provide both python2/3 or alias python2=python to let your system choose where to route the command.

Install fuga (cli)

$ # (on your own machine)
$ pip install fuga

Install fuga templates

Fuga powers cookiecutter to offer various templates/boilerplates for fuga experiments. You need to install it to your environment before using fuga.

$ git clone \
$ git clone \

Create Cloud Composer Environment

Creating new Cloud Composer Environment is not supported by the tool at the moment.

Install fuga to Composer Environment

You need to install fuga to Compooser Environment if you want to use fuga airflow utilities.

Initialize Fuga environment

Fuga needs to know which GCP project and GCS bucket to use with it. fuga environment init command with let you choose one or create new for each.

$ fuga environment init
# follow instructions

Create your experiment

$ fuga experiment new my_experiment

Deploy your experiment

$ cd my_experiment
$ fuga experiment deploy

(optional) Create your implementation for KubernetesPodOperator

If you want to use an operator with external dependency which is not able to be resolved using just PyPI packages, (e.g. MeCab) you may need to use KuberenetesPodOperator.

Fuga supports generating a scaffold for KuberenetesPodOperator and deploy(and rollback(WIP)) it on Google Cloud Container Registry.

$ cd my_experiment
$ fuga pod-operator new my_pod_operator
$ fuga pod-operator deploy my_pod_operator

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fuga, version 0.1.7
Filename, size File type Python version Upload date Hashes
Filename, size fuga-0.1.7-py2.py3-none-any.whl (20.9 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page