VDK Lineage
VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured destination. The lineage data is send using OpenLineage standard
At POC level currently.
Currently, lineage data is collected
- For each data job run/execution both start and end events including the status of the job (failed/succeeded)
- For each execute query we collect input and output tables.
TODOs:
- Collect status of the SQL query (failed, succeeded)
- Create parent /child relationship between sql event and job run event to track them better (single job can have multiple queries)
- Non-SQL lineage (ingest, load data,etc)
- Extend support for all queries
- provide more information using facets – op id, job version,
- figure out how to visualize parent/child relationships in Marquez
- Explore openlineage.sqlparser instead of sqllineage library as alternative
Usage
pip install vdk-lineage
And it will start collecting lineage from job and sql queries.
To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:
export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stop
Build and testing
In order to build and test a plugin go to the plugin directory and use ../build-plugin.sh script to build it
1 maintainer
Release files for vdk-lineage 0.3.1431637373
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| vdk_lineage-0.3.1431637373.tar.gz | 12.7 kB | Details |
Release files / vdk_lineage-0.3.1431637373.tar.gz
| Download URL | vdk_lineage-0.3.1431637373.tar.gz |
|---|---|
| Size | 12.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2b568532f5c683548732b77c78157238c02ac6195a2fb27bcb465c365a346a0f
|
|
BLAKE2b-256 checksum How to use checksums |
03d123ba9bfb25b87f12560cc2ceedb22c12623d14f1e23b42b90b4c84369009
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.14
|
