Pyplatform is a data analytics platform built around Google BigQuery. This package provides wrapper functions for interacting with cloud services and creating data pipelines using Google Cloud, Microsoft Azure, O365, and Tableau Server as source and destination.
The platorm architecture:
- enables fast and scalable SQL datawarehousing service
- abstracts away the infrastuture by builiding data pipelines with serverless compute solutions in python runtime environments
- simplifies development environment by using jupyter lab as the main tool
Installation
pip install pyplatform
Setting up development environment
git clone https://github.com/mhadi813/pyplatform
cd pyplatform
conda env create -f pyplatform_dev.yml
Authentication and environment variables
Credential file path can be set a environment varible in conda env activation script or bash profiles. Please reference conda documentation for enviroment variables and environment activation script
import os
# if env activation script not created: update path to credential files
# see ``secrets`` folder for credential tamplates
# see functions ``docstring`` for authentication methods when calling a function
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = './secrets/dummy_gcp_service_account_credentials.json'
os.environ["AZURE_CREDENTIALS"]= './secrets/dummy_ms_azure_credentials.json'
os.environ['TABLEAU_SERVER_CREDENTIALS']='./secrets/dummy_tableau_server_credentials.json'
os.environ['PIVOTAL_CREDENTIALS']='./secrets/dummy_pivotal_credentials.json'
os.environ['DATASET'] = 'default_bigquery_dataset_name'
os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'
Usage
common usage patterns:
- Http sources
- On-prem sources with VPN requirement
- Bigquery integration with Azure Logic Apps
- Event driven ETL process
- Streaming pipelines
Exploring the modules
from pyplatform.common import *
show_me()
import pyplatform as pyp
show_me(pyp)
1 maintainer
Metadata
1 maintainer
Release files for pyplatform 2020.12.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| pyplatform-2020.12.1.tar.gz | 445.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pyplatform-2020.12.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 460.0 kB
Release files / pyplatform-2020.12.1.tar.gz
| Download URL | pyplatform-2020.12.1.tar.gz |
|---|---|
| Size | 445.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
30763c1f3d78c50b605024d8c6740a47f91e2d77ff37bc0be1ad6251ed636d10
|
|
BLAKE2b-256 checksum How to use checksums |
260134cd76149d7b0288f919cf2dfbdf671a70e7adee0f4ce29ab8332cd6c3f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.6
|
Release files / pyplatform-2020.12.1-py3-none-any.whl
| Download URL | pyplatform-2020.12.1-py3-none-any.whl |
|---|---|
| Size | 14.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
16c8b7145ba3e71dfc7ad466f68e3b9aa0d340f83d8905c5e88aa36fbe26b9f3
|
|
BLAKE2b-256 checksum How to use checksums |
9973ed7b49ef7677ad481a32b409cc484059f847e4fe70ed8634f04cbed8b7d3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.6
|
Security for pyplatform 2020.12.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
