VDK Lineage

VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured destination. The lineage data is send using OpenLineage standard

At POC level currently.

Currently, lineage data is collected

For each data job run/execution both start and end events including the status of the job (failed/succeeded)
For each execute query we collect input and output tables.

TODOs:

Collect status of the SQL query (failed, succeeded)
Create parent /child relationship between sql event and job run event to track them better (single job can have multiple queries)
Non-SQL lineage (ingest, load data,etc)
Extend support for all queries
provide more information using facets – op id, job version,
figure out how to visualize parent/child relationships in Marquez
Explore openlineage.sqlparser instead of sqllineage library as alternative

Usage

pip install vdk-lineage

And it will start collecting lineage from job and sql queries.

To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:

export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stop

Build and testing

In order to build and test a plugin go to the plugin directory and use ../build-plugin.sh script to build it

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vdk-lineage-0.3.582131318.tar.gz (10.0 kB view details)

Uploaded Jul 7, 2022 Source

File details

Details for the file vdk-lineage-0.3.582131318.tar.gz.

File metadata

Download URL: vdk-lineage-0.3.582131318.tar.gz
Upload date: Jul 7, 2022
Size: 10.0 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/4.0.1 CPython/3.7.13

File hashes

Hashes for vdk-lineage-0.3.582131318.tar.gz
Algorithm	Hash digest
SHA256	`88f6e17ce9d48eefa96f760eda3407dcb2ed8083e47a2e9d8cfda008e828e3e4`
MD5	`2e4759ae4bf8844df5aa36dddfa2d05b`
BLAKE2b-256	`ed011f468fedced233a2481d5cf488bdb6b8a86fe9ea3f8c59696dd23c4de7be`