data-rentgen 0.5.1

Data.Rentgen REST API + Kafka consumer

What is Data.Rentgen?

Data.Rentgen is a Data Motion Lineage service, compatible with OpenLineage specification.

Currently we support consuming lineage from:

• Apache Spark

• Apache Airflow

• Apache Hive

• Apache Flink

• dbt

• SyncMaster (proprietary integration, part of MWS Data Bridge)

• StarRocks (proprietary integration, part of MWS Data Engine)

Note: service is under active development, so API can be unstable.

Goals

• Collect lineage events produced by OpenLineage clients & integrations.

• Store operation-grained events for better detalization.

• Provide API for fetching both job/run ↔ dataset lineage and dataset ↔ dataset lineage.

Features

• Support consuming large amounts of lineage events, use Apache Kafka as event buffer.

• Store data in tables partitioned by event timestamp, to speed up lineage graph resolution.

• Lineage graph is build with user-specified time boundaries.

• Lineage graph can be build with different granularity. e.g. merge all individual Spark commands into Spark applicationId or Spark applicationName.

• Column-level lineage support.

• Authentication support.

Non-goals

• This is not a Data Catalog. DataRentgen doesn’t track dataset schema change, owner and so on. Use Datahub or OpenMetadata instead.

• Static Data Lineage like view → table is not supported.

Limitations

• OpenLineage have integrations with Trino, Debezium and some other lineage sources. DataRentgen support may be added later.

• DataRentgen parses only limited set of OpenLineage facets, and doesn’t store custom facets. This can be changed in future.

Documentation

See https://data-rentgen.readthedocs.io/

Screenshots

Lineage graph

Dataset-level lineage graph

Dataset column-level lineage graph

Job-level lineage graph

Run-level lineage graph

Hierarchy graph

Datasets

Runs

Spark application

Spark run

Spark command

Hive query

Airflow DagRun

Airflow TaskInstance