Prophecy Lineage Extractor Documentation
Description
The Prophecy Lineage Extractor is a tool to extract lineage information from Prophecy projects and pipelines. It allows users to specify a project, pipeline, and branch, and outputs the extracted lineage to a specified directory. Optional features include email notifications.
Usage
python -m prophecy_lineage_extractor --project-id <PROJECT_ID> --pipeline-id <PIPELINE_ID> --output-dir <OUTPUT_DIRECTORY> [--send-email] [--branch <BRANCH_NAME>] [--run-for-all]
- We must need to set these env variables PROPHECY_URL and PROPHECY_PAT
Arguments
Required Arguments
-
--project-id
- Type: str
- Description: Prophecy Project ID.
- Required: Yes
-
--pipeline-id
- Type: str
- Description: Prophecy Pipeline ID.
- Required: Yes / Optional if using
knowledge-graphtype reader.
-
--output-dir
- Type: str
- Description: Output directory inside the project where lineage files will be stored.
- Required: Yes
Optional Arguments
-
--reader:
- Type: str
- Description: Reading adapter to use
- Spark Lineage (
lineage) or- Note that
pipeline-idis mandatory argument for this method at the moment.
- Note that
- SQL Knowledge Graph (
knowledge-graph)
- Spark Lineage (
-
--writer:
- Type: str
- Description: Data Format to write to from among:
- Excel Files (xlxs sheet)
- We save a sheet with name lineage__(<optional_pipeline-ids>.xlsx) will be created in
<output_dir>. - For each pipeline mentioned in the query, an Excel sheet is created. If
run-for-allis used, a sheet per pipeline is created. - If
run-for-allflag is used, an additionalOverall Projectsheet will be created. NOTE that it
- We save a sheet with name lineage__(<optional_pipeline-ids>.xlsx) will be created in
- Openlineage Format
- We save Dummy Run Events JSON files in OpenLineage format in the output-dir/ folder
- We attempt to make an API Call to an Openlineage compatible frontend like Marquez or Datahub.
- We support Column Level Lineage as well as Project Level Lineage via this method.
- Excel Files (xlxs sheet)
-
--run-for-all
- Type: boolean flag
- Description: If Specified, a Project level Lineage Excel file is created as an Overall Project.
-
--send-email
- Type: flag
- Description: If specified, sends an email with the generated lineage report to ENV variable RECEIVER_EMAIL.
- We must set following Env variables for this option if passed
- SMTP_HOST
- SMTP_PORT
- SMTP_USERNAME
- SMTP_PASSWORD
- RECEIVER_EMAIL
-
--branch
- Type: str
- Description: Branch to run the lineage extractor on.
- Default: default branch in Prophecy, generally 'main or master'
Running
- Please run extractor as following, it needs env variables
- we Only need to set SMTP creds if we plan to pass
--send-emailargument
export PROPHECY_URL=https://app.prophecy.io
export PROPHECY_PAT=${{ secrets.PROPHECY_PAT }}
# These are needed if you using --send-email option
export SMTP_HOST=smtp.gmail.com
export SMTP_PORT=587
export SMTP_USERNAME=${{ secrets.SMTP_USERNAME }}
export SMTP_PASSWORD=${{ secrets.SMTP_PASSWORD }}
export RECEIVER_EMAIL=ashish@prophecy.io
python -m prophecy_lineage_extractor --project-id 36587 --pipeline-id 36587/pipelines/customer_orders_demo --send-email --branch dev
Github Action Guide
-
This extactor can be setup in Github Action of a Prophecy project to get email of lineage on every commit to main
-
Following is a sample of github action we can use on default branch Github Action default branch
-
Following is a sample of github action we can use on custom branch Github Action custom branch
Gitlab Action Guide
- Following is a sample of gitlab action we can use on a branch Gitlab Action guide
- Note—we need to create gitlab CI/CD variables(secrets) for using them in our YML file, ex. SMTP_USER etc.
- additionally, we will also need to setup an ACCESS_TOKEN to allow the JOB to commit if commit is enabled.
Metadata
Release files for prophecy-lineage-extractor 0.21.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| prophecy_lineage_extractor-0.21.1.tar.gz | 25.2 kB | Without provenance | Details for prophecy_lineage_extractor-0.21.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| prophecy_lineage_extractor-0.21.1-py3-none-any.whl | Python 3 | none | any | Details for prophecy_lineage_extractor-0.21.1-py3-none-any.whl |
Total release size: 56.4 kB
Release files / prophecy_lineage_extractor-0.21.1.tar.gz
| Download URL | prophecy_lineage_extractor-0.21.1.tar.gz |
|---|---|
| Size | 25.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bf0c0e76a7d892af36ee28416803bda78a0ba90f1fca2fef31e736c81c423baa
|
|
BLAKE2b-256 checksum How to use checksums |
e66d4ac8cc34fd5b82abffb4c15f4931e607488d65c5c69dec70df5ab863bd73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.13
|
Release files / prophecy_lineage_extractor-0.21.1-py3-none-any.whl
| Download URL | prophecy_lineage_extractor-0.21.1-py3-none-any.whl |
|---|---|
| Size | 31.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
36d019f9418c3afac42c767f29598d5fd2f5668a59b0908b35149b778487cf0c
|
|
BLAKE2b-256 checksum How to use checksums |
4631b366e7b1a3e88156511f5f6834aee164659552eeaf94f5332b7985f0bd17
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.13
|
Security for prophecy-lineage-extractor 0.21.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.