Skip to main content

Summit Public Schools Snowflake warehouse for use in Airflow

Project description

spswarehouse_airflow

This package is a clone of spswarehouse, adapted to be used in Airflow. Rather than using a credentials.py file, it pulls credentials from Airflow's Connections.

Current Version

Currently, only the Warehouse-related functionality has been adapted. Google Drive, Google Sheets, and Google Slides are not yet supported.

Usage

The main difference in this packages is that you do not import Warehouse directly from the package. Instead, you will import the function create_warehouse, and then call that function to create the warehouse connection when you need it. After completing the Warehouse work, you need to manually close the Warehouse connection.

At the top of your DAG file:

from spswarehouse_airflow.warehouse import create_warehouse
from spswarehouse_airflow.table_utils import *

In a function that is fed to a PythonOperator:

Warehouse = create_warehouse()
<Whatever it is you need to do in the warehouse>
Warehouse.close()

Project details


Release history Release notifications

This version

0.0.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spswarehouse-airflow, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size spswarehouse_airflow-0.0.2.tar.gz (7.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page