Skip to main content

Data engineering & Data science Pipeline Framework

Project description

PyGyver

PyGyver is a user-friendly python package for data integration and manipulation.

Named after MacGyver, title character in the TV series MacGyver, and Python, the main language used in the repository.

Installation

PyPi

PyGyver is available on PyPi.

pip install pygyver

Setup

Most APIs requires access token files to authentificate and perform tasks such as creating or deleting objects. Those files need to be generated prior to using pygyver and stored in the environment you are executing your code against. The package make use of environment variables, and some of the below might need be supplied in your environment:

# Access token path
GOOGLE_APPLICATION_CREDENTIALS=path_to_google_access_token.json
FACEBOOK_APPLICATION_CREDENTIALS=path_to_facebook_access_token.json

# Default values
BIGQUERY_PROJECT=your-gcs-project
GCS_PROJECT=your-gcs-project
GCS_BUCKET=your-gcs-bucket

# Optional
PROJECT_ROOT=path_to_where_your_code_lives

Modules

PyGyver is structured around several modules available in the etl folder. Here is a summary table of those modules:

Module name Descrition Documentation
dw Perform task against the Google Cloud BigQuery API dw.md
facebook Perform task against the Facebook Marketing API facebook.md
gooddata Perform task against the GoodData API -
gs Perform task against the Google Sheet API -
lib Store utilities used by other modules -
pipeline Utility to build data pipelines via YAML definition pipeline.md
prep Data transformation - ML pipelines -
storage Perform task against the AWS S3 and Google Cloud Storage API storage.md
toolkit Sets of tools for data manipulation -

In order to load BigQueryExecutor from the dw module, you can run:

from pygyver.etl.dw import BigQueryExecutor

Contributing

To get started...

Step 1

  • 👯 Clone this repo to your local machine using git@github.com:madedotcom/pygyver.git

Step 2

  • HACK AWAY! 🔨🔨🔨

The team follows TDD to develop new features on pygyver. Tests can be found in pygyver/tests.

Step 3

  • 🔃 Create a new pull request and request review from team members. Where applicable, a test should be added with the code change.

FAQ

  • How to release a new version to PyPi?
    1. Merge your changes to master branch
    2. Create a new release using https://github.com/madedotcom/pygyver/releases

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pygyver-0.1.1.42.tar.gz (63.4 kB view hashes)

Uploaded Source

Built Distribution

pygyver-0.1.1.42-py3-none-any.whl (52.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page