Skip to main content

Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.

Project description

Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.

the platorm:

  • provides fast, scalable and reliable SQL database
  • abstracts away the infrastuture by builiding data pipelines with serverless compute solutions in python runtime environments
  • simplifies development environment by using jupyter lab as the main tool

Installation

pip install pyplatform

Setting up development environment

git clone https://github.com/mhadi813/pyplatform
cd pyplatform
conda env create -f pyplatform_dev.yml

Environment variables

import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/default_service_account.json'
os.environ['DATASET'] = 'default_bigquery_dataset_name'
os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'

Usage

common data pipeline architectures:

- Http sources

- On-prem servers

- Bigquery integration with Azure Logic Apps

- Event driven ETL process

- Streaming pipelines

Exploring modules

import pyplatform as pyp
pyp.show_me()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyplatform-0.1.4.tar.gz (525.6 kB view hashes)

Uploaded Source

Built Distribution

pyplatform-0.1.4-py3-none-any.whl (34.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page