prophecy-lineage-extractor 0.20.0

No project description provided

Prophecy Lineage Extractor Documentation

Description

The Prophecy Lineage Extractor is a tool to extract lineage information from Prophecy projects and pipelines. It allows users to specify a project, pipeline, and branch, and outputs the extracted lineage to a specified directory. Optional features include email notifications.

---

Usage

python -m prophecy_lineage_extractor --project-id <PROJECT_ID> --pipeline-id <PIPELINE_ID> --output-dir <OUTPUT_DIRECTORY> [--send-email] [--branch <BRANCH_NAME>] [--recursive_extract <true/false>] [--run_for_all <true/false>]

• We must need to set these env variables PROPHECY_URL and PROPHECY_PAT

---

Arguments

Required Arguments

• --project-id

  • Type: str
  • Description: Prophecy Project ID.
  • Required: Yes

• --pipeline-id

  • Type: str
  • Description: Prophecy Pipeline ID.
  • Required: Yes

• --output-dir

  • Type: str
  • Description: Output directory inside the project where lineage files will be stored.
  • Required: Yes

Optional Arguments

• --run_for_all

  • Type: boolean flag
  • Description: If Specified, a Project level Lineage Excel file is created as an Overall Project.

• --recursive-extract

  • Type: boolean flag
  • Description: If Specified, for any column from a source, which has been changed upstream, a recursive search for the same is displayed.

• --send-email

  • Type: flag
  • Description: If specified, sends an email with the generated lineage report to ENV variable RECEIVER_EMAIL.
  • We must set following Env variables for this option if passed
    • SMTP_HOST
    • SMTP_PORT
    • SMTP_USERNAME
    • SMTP_PASSWORD
    • RECEIVER_EMAIL

• --branch

  • Type: str
  • Description: Branch to run the lineage extractor on.
  • Default: default branch in Prophecy, generally 'main or master'

---

Running

• Please run extractor as following, it needs env variables
• we Only need to set SMTP creds if we plan to pass --send-email argument

export PROPHECY_URL=https://app.prophecy.io
export PROPHECY_PAT=${{ secrets.PROPHECY_PAT }}

# These are needed if you using --send-email option
export SMTP_HOST=smtp.gmail.com
export SMTP_PORT=587
export SMTP_USERNAME=${{ secrets.SMTP_USERNAME }}
export SMTP_PASSWORD=${{ secrets.SMTP_PASSWORD }}
export RECEIVER_EMAIL=[e-mail]

python -m prophecy_lineage_extractor --project-id 36587 --pipeline-id 36587/pipelines/customer_orders_demo --send-email --branch dev

---

Github Action Guide

• This extactor can be setup in Github Action of a Prophecy project to get email of lineage on every commit to main

• Following is a sample of github action we can use on default branch Github Action default branch

• Following is a sample of github action we can use on custom branch Github Action custom branch

---

Gitlab Action Guide

• Following is a sample of gitlab action we can use on a branch Gitlab Action guide
• Note—we need to create gitlab CI/CD variables(secrets) for using them in our YML file, ex. SMTP_USER etc.
• additionally, we will also need to setup an ACCESS_TOKEN to allow the JOB to commit if commit is enabled.