pyplatform 2020.7.2

Pyplatform provides wrapper functions for using Google BigQuery as datawarehouse and creating data pipelines involving Google Cloud, Microsoft Azure, O365, and Tableau Server as source and destination.

Pyplatform provides wrapper functions for using Google BigQuery as datawarehouse and creating data pipelines involving Google Cloud, Microsoft Azure, O365, and Tableau Server as source and destination.

the platorm architecture:

• enables fast and scalable SQL datawarehousing service
• abstracts away the infrastuture by builiding data pipelines with serverless compute solutions in python runtime environments
• simplifies development environment by using jupyter lab as the main tool

Installation

pip install pyplatform

Setting up development environment

git clone https://github.com/mhadi813/pyplatform
cd pyplatform
conda env create -f pyplatform_dev.yml

Environment variables

import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/default_service_account.json'
os.environ['DATASET'] = 'default_bigquery_dataset_name'
os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'

Usage

common data pipeline architectures:

- Http sources

- On-prem servers

- Bigquery integration with Azure Logic Apps

- Event driven ETL process

- Streaming pipelines

Exploring modules

import pyplatform as pyp
pyp.show_me()