pyplatform 0.1.3

Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.

Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.

the platorm: - provides fast, scalable and reliable SQL database -
abstracts away the infrastuture by builiding data pipelines with
serverless compute solutions in python runtime environments - simplifies
development environment by using jupyter lab as the main tool

Installation
------------

.. code:: python

   pip install pyplatform

Setting up development environment
----------------------------------

::

   git clone https://github.com/mhadi813/pyplatform
   cd pyplatform
   conda env create -f pyplatform_dev.yml

`Environment variables <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#saving-environment-variables>`__

.. code:: python

import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/default_service_account.json' os.environ['DATASET'] = 'default_bigquery_dataset_name' os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'

Usage

common data pipeline architectures:

• Http sources

- On-prem servers

• Bigquery integration with Azure Logic Apps

- Event driven ETL process
~~~~~~~~~~~~~~~~~~~~~~~~~~

- Streaming pipelines
~~~~~~~~~~~~~~~~~~~~~

Exploring modules
-----------------

.. code:: python


   import pyplatform as pyp
   pyp.show_me()