Google Cloud Storage and Azure Storage functions for working with unstructured data
Project description
Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.
the platorm:
- provides fast, scalable and reliable SQL database solution
- abstracts away the infrastuture by builiding data pipelines with serverless compute solutions in python runtime environments
- simplifies development environment by using jupyter lab as the main tool
Installation
pip install pyplatform
Setting up development environment
git clone https://github.com/mhadi813/pyplatform
cd pyplatform
conda env create -f pyplatform_dev.yml
Environment variables
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/default_service_account.json'
os.environ['DATASET'] = 'default_bigquery_dataset_name'
os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'
Usage
common data pipeline architectures:
- Http sources
- On-prem servers
- Bigquery integration with Azure Logic Apps
- Event driven ETL process
- Streaming pipelines
Exploring modules
import pyplatform as pyp
pyp.show_me()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for pyplatform-datalake-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 708e8c1ae78b94fe6270e88fb81ee7328206d18c60b37e4b66b4192c2407cb41 |
|
MD5 | 6c0cc52da7c03fd419924d49f8ebabe9 |
|
BLAKE2b-256 | 359e7c458173b6d7faf0cce93f84887926b2252c66e0797d64fe2a150c7385f7 |
Close
Hashes for pyplatform_datalake-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f40eb5c4afb386ea31c3a2843646a11c5950c48ab0a0de2fd4e1ce568cff3b45 |
|
MD5 | 956c3e6de6c9882211bdff248e003c6f |
|
BLAKE2b-256 | 5f8e5873598641f6ef06c2f201514ad1fbdb32a68eecb0f4d2b92e7eb5a685a8 |