Skip to main content

Summit Public Schools Snowflake warehouse for use in Airflow

Project description

spswarehouse_airflow

This package is a clone of spswarehouse, adapted to be used in Airflow. Rather than using a credentials.py file, it pulls credentials from Airflow's Connections.

Usage

Snowflake Access

The main difference in this packages is that you do not import Warehouse directly from the package. Instead, you will import the function create_warehouse, and then call that function to create the warehouse connection when you need it. After completing the Warehouse work, you need to manually close the Warehouse connection.

At the top of your DAG file:

from spswarehouse_airflow.warehouse import create_warehouse
from spswarehouse_airflow.table_utils import *

In a function that is fed to a PythonOperator:

Warehouse = create_warehouse()
<Whatever it is you need to do in the warehouse>
Warehouse.close()

Google Access

Similar to Snowflake, rather than importing GoogleSheets or GoogleDrive, import the relevant create function.

from spswarehouse_airflow.googlesheets import create_sheets
from spswarehouse_airflow.googledrive import create_drive
from spswarehouse_airflow.googleslides import create_slides
GoogleSheets = create_sheets()
GoogleDrive = create_drive()
GoogleSlides = create_slides()

Default service account e-mail to share your files with is airflow@dops-special-projects.iam.gserviceaccount.com

If you are not using google_cloud_default as the connection name for where you are storing the credentials, pass the conn_id to the create function. E.g., GoogleSheets = create_sheets(custom_conn_id)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spswarehouse_airflow-0.5.1.tar.gz (22.1 kB view details)

Uploaded Source

File details

Details for the file spswarehouse_airflow-0.5.1.tar.gz.

File metadata

  • Download URL: spswarehouse_airflow-0.5.1.tar.gz
  • Upload date:
  • Size: 22.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.9

File hashes

Hashes for spswarehouse_airflow-0.5.1.tar.gz
Algorithm Hash digest
SHA256 faf74425a8636e8843f8c96159dfbe0a3a767abc9db630b7b44d40d13276a3e1
MD5 d1c970109f042326d5db0788b04ca027
BLAKE2b-256 869e39f2f4beb016907c4b49c57625f8ce7c23995864ea4f0ea56b9082071c85

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page