Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.
Project description
Pyplatform is a data analytics platform architeture built around Google BigQuery in a hybrid cloud environment.
the platorm: - provides fast, scalable and reliable SQL database -
abstracts away the infrastuture by builiding data pipelines with
serverless compute solutions in python runtime environments - simplifies
development environment by using jupyter lab as the main tool
Installation
------------
.. code:: python
pip install pyplatform
Setting up development environment
----------------------------------
::
git clone https://github.com/mhadi813/pyplatform
cd pyplatform
conda env create -f pyplatform_dev.yml
`Environment variables <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#saving-environment-variables>`__
.. code:: python
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/default_service_account.json' os.environ['DATASET'] = 'default_bigquery_dataset_name' os.environ['STORAGE_BUCKET'] = 'default_storage_bucket_id'
Usage
common data pipeline architectures:
- Http sources
- On-prem servers
- Bigquery integration with Azure Logic Apps
- Event driven ETL process
~~~~~~~~~~~~~~~~~~~~~~~~~~
- Streaming pipelines
~~~~~~~~~~~~~~~~~~~~~
Exploring modules
-----------------
.. code:: python
import pyplatform as pyp
pyp.show_me()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyplatform-0.1.3.tar.gz
(525.6 kB
view hashes)
Built Distribution
pyplatform-0.1.3-py3-none-any.whl
(34.6 kB
view hashes)
Close
Hashes for pyplatform-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b2100243182162837d2a99c35dadb4ed13b2d851d84f3c45953b466d96016fee |
|
MD5 | 4b563cf15cf8d05cc1a8476601ea2f7d |
|
BLAKE2b-256 | cef74687729171625de490486a295b304be770d056ef58acf098b0a571fdab53 |