A airflow_sops secrets backend with sops
Project description
Airflow SOPS Secrets Backend for GCP KMS
This packages enables Airflow to pull connections and variables from files in GCP bucket that are encrypted by SOPS using GCP KMS.
Configure Airflow
Add following to airflow.cfg.
[secrets]
backend = airflow_sops.secrets_backend.GcsSopsSecretsBackend
backend_kwargs = {"project_id": "your-project-id"}
Available parameters to backend_kwargs:
- project_id: Optional. GCP project id where the GCS bucket which holds the encrypted connections/variables files reside.,
- bucket_name: Optional. If not submitted tries retrieving from Composer GCS_BUCKET environment variable
- connections_prefix. Optional. Default is "sops/connections". The folder in GCS bucket that holds encrypted connections.
- variables_prefix: Optional. Default is "sops/variables". The folder in GCS bucket that holds encrypted variables.,
- encrypted_file_ext: Optional. Default is "enc". The file extension for encrypted sops files. The format is <connection_id or variable_key>.<encrypted_file_ext>.yaml
- ignore_mac: Optional. Default is True. Ignores file checksum when true.
GCP Config
locals {
gcp_project_id = "your-project-id"
service_account_name = "your-composer-service-account-name"
}
resource "google_service_account" "composer" {
account_id = local.service_account_name
display_name = local.service_account_name
project = local.gcp_project_id
}
resource "google_project_iam_member" "composer_worker" {
project = local.gcp_project_id
role = "roles/composer.worker"
member = "serviceAccount:${google_service_account.composer.email}"
}
resource "google_kms_key_ring" "secrets" {
name = local.gcp_project_id
location = "europe-west1"
project = local.gcp_project_id
}
resource "google_kms_crypto_key" "secrets_sops" {
name = "secrets_sops"
key_ring = google_kms_key_ring.secrets.id
rotation_period = "7776000s" // 90 days
}
resource "google_kms_crypto_key_iam_member" "composer_sops_decrypter" {
crypto_key_id = google_kms_crypto_key.secrets_sops.id
role = "roles/cloudkms.cryptoKeyDecrypter"
member = "serviceAccount:${google_service_account.composer.email}"
}
# some mandatory attributes omitted
resource "google_composer_environment" "composer" {
name = "your-composer-environment-name"
region = "europe-west1"
project = local.gcp_project_id
config {
software_config {
airflow_config_overrides = {
secrets-backend = "airflow_sops.secrets_backend.GcsSopsSecretsBackend"
}
pypi_packages = {
airflow-secrets-sops = "==0.0.1"
}
}
node_config {
service_account = google_service_account.composer.email
}
}
}
SOPS
Install SOPS. Encrypt files using GCP KMS and upload to GCP bucket sops/connections directory
export KMS_PATH=$(gcloud kms keys list --location europe-west1 --keyring your-keyring --project your-gcp-project | awk 'FNR == 2 {print $1}')
sops --encrypt --encrypted-regex '^(password|extra)$' --gcp-kms $KMS_PATH some-connection.yaml > some-connection.enc.yaml
Setup
python -m venv .venv
source .venv/bin/activate
pip config set --site global.extra-index-url https://pypi.org/simple
pip install -r requirements.txt
Test
pip install . airflow-sops-secrets-backend[test]
pytest
Build
pip install airflow-sops-secrets-backend[dev]
python -m build
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-sops-secrets-backend-0.0.2.tar.gz
.
File metadata
- Download URL: airflow-sops-secrets-backend-0.0.2.tar.gz
- Upload date:
- Size: 12.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a2c3e5db00b9447434929de4e44751832de47c2c24b4ecaf8c3fc7fd5622b39 |
|
MD5 | 3e118849c4cda10845186c8a7c481be0 |
|
BLAKE2b-256 | 040228e02c1c14a77a409a1ca6c67d70309106f1657f9df95ccba3d5cdaf2860 |
File details
Details for the file airflow_sops_secrets_backend-0.0.2-py3-none-any.whl
.
File metadata
- Download URL: airflow_sops_secrets_backend-0.0.2-py3-none-any.whl
- Upload date:
- Size: 9.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 212f2db87acb432261bdf0b97462e0a8817968918135ae6e577c6346f362b040 |
|
MD5 | f433369beffa784fcbef6a6cbc8f775e |
|
BLAKE2b-256 | 145ea6873ec784325e4eeaeaca7aad61991212030a804f1223a8344ded019af0 |