spswarehouse_airflow
This package is a clone of spswarehouse, adapted to be used in Airflow. Rather than using a credentials.py file, it pulls credentials from Airflow's Connections.
Usage
Snowflake Access
The main difference in this packages is that you do not import Warehouse directly from the package. Instead, you will import the function create_warehouse, and then call that function to create the warehouse connection when you need it. After completing the Warehouse work, you need to manually close the Warehouse connection.
At the top of your DAG file:
from spswarehouse_airflow.warehouse import create_warehouse
from spswarehouse_airflow.table_utils import *
In a function that is fed to a PythonOperator:
Warehouse = create_warehouse()
<Whatever it is you need to do in the warehouse>
Warehouse.close()
Google Access
Similar to Snowflake, rather than importing GoogleSheets or GoogleDrive, import the relevant create function.
from spswarehouse_airflow.googlesheets import create_sheets
from spswarehouse_airflow.googledrive import create_drive
from spswarehouse_airflow.googleslides import create_slides
GoogleSheets = create_sheets()
GoogleDrive = create_drive()
GoogleSlides = create_slides()
Default service account e-mail to share your files with is airflow@dops-special-projects.iam.gserviceaccount.com
If you are not using google_cloud_default as the connection name for where you are storing the credentials, pass the conn_id to the create function. E.g., GoogleSheets = create_sheets(custom_conn_id)
2 maintainers
Release files for spswarehouse-airflow 0.6.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spswarehouse_airflow-0.6.3.tar.gz | 23.2 kB | Details |
Release files / spswarehouse_airflow-0.6.3.tar.gz
| Download URL | spswarehouse_airflow-0.6.3.tar.gz |
|---|---|
| Size | 23.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f012b77585930ca227b7eaf4abc0623c6976407182e2ddef803bc451a1c7149
|
|
BLAKE2b-256 checksum How to use checksums |
b87cacebfcd7c0313ffc69f8f5fcaeb419bf6b58fdc2b7a5e1acc710f39a6ece
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.9
|
