Data engineering & Data science Pipeline Framework
PyGyver is a user-friendly python package for data integration and manipulation.
Named after MacGyver, title character in the TV series MacGyver, and Python, the main language used in the repository.
PyGyver is available on PyPi.
pip install pygyver
Most APIs requires access token files to authentificate and perform tasks such as creating or deleting objects. Those files need to be generated prior to using
pygyver and stored in the environment you are executing your code against. The package make use of environment variables, and some of the below might need be supplied in your environment:
# Access token path GOOGLE_APPLICATION_CREDENTIALS=path_to_google_access_token.json FACEBOOK_APPLICATION_CREDENTIALS=path_to_facebook_access_token.json # Default values BIGQUERY_PROJECT=your-gcs-project GCS_PROJECT=your-gcs-project GCS_BUCKET=your-gcs-bucket # Optional PROJECT_ROOT=path_to_where_your_code_lives
PyGyver is structured around several modules available in the
etl folder. Here is a summary table of those modules:
||Perform task against the Google Cloud BigQuery API||dw.md|
|Perform task against the Facebook Marketing API||facebook.md|
||Perform task against the GoodData API||-|
||Perform task against the Google Sheet API||-|
||Store utilities used by other modules||-|
||Utility to build data pipelines via YAML definition||pipeline.md|
||Data transformation - ML pipelines||-|
||Perform task against the AWS S3 and Google Cloud Storage API||storage.md|
||Sets of tools for data manipulation||-|
In order to load
BigQueryExecutor from the
dw module, you can run:
from pygyver.etl.dw import BigQueryExecutor
To get started...
- 👯 Clone this repo to your local machine using
- HACK AWAY! 🔨🔨🔨
The team follows TDD to develop new features on
Tests can be found in
- 🔃 Create a new pull request and request review from team members. Where applicable, a test should be added with the code change.
- How to release a new version to PyPi?
- Merge your changes to
- Create a new release using
- Merge your changes to
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pygyver-0.1.1.17.tar.gz (49.7 kB)||File type Source||Python version None||Upload date||Hashes View|