dockflow 2.0.0

Easily deploy Airflow for local development

Dockflow CLI

This package requires docker to be installed and running on your machine.

Getting started

1. Install dockflow:
   pip install dockflow

2. Authenticate docker to pull from private GCR repository:
   2.1. First make sure that your gcloud is authenticated using gcloud auth application-default login --impersonate-service-account=<service-account>.
   2.2. Ensure you are in the correct project using gcloud config set project <project name>
   2.3. Use gcloud auth configure-docker <location>-docker.pkg.dev to configure gcloud as the credential helper for the Artifact Registry domain associated with your repository’s location. For example, gcloud auth configure-docker europe-west1-docker.pkg.dev.

Quickstart

1. Ensure that your docker file sharing settings allows access to your development directory.
2. Navigate to the root directory of your dags folder.
3. Use dockflow config and enter your container repo url excluding the version.
   • This will save the url in a config file.
   • Eg. <location>-docker.<domain>/<project-id>/<repo-id>/<image>
   • If you get permission errors, Ensure that you have artifact reader permission on this registry.
   • This should only be used if the container repo changes.
4. If the image version tag is not composer-2.9.8-airflow-2.9.3 specify the tag using dockflow start -iv <version>. Alternatively, use the --check-images flag to list the available images at the provided container repo.
5. Use dockflow start (This will mount the dag folder and start Airflow).
6. Use the UI to add connections.
   • Admin -> Connections -> Create
   • If you need to connect to a GCP resource, provide values for the following fields:
     • Project Id, e.g. research-se-de
     • Credential configuration file, located at /usr/local/airflow/gcp_credentials.json
     • Impersonation chain, e.g. [e-mail]
7. Use dockflow refresh to refresh the configs cache or to bundle configs.
8. Remember to use dockflow stop to shut down the instance to save local machine resources.
   • The state will be persisted in the same directory as the dags folder.
9. To stop and remove the container use dockflow stop --rm

CloudSQL Proxy

docker run -d \
  -v <PATH_TO_KEY_FILE>:/config \
  -p 127.0.0.1:5432:5432 \
  --network='dockflow' \
  --name='cloudsql' \
  gcr.io/cloudsql-docker/gce-proxy:1.17 /cloud_sql_proxy \
  -instances=<INSTANCE_CONNECTION_NAME>=tcp:0.0.0.0:5432 -credential_file=/config