Data.Rentgen logo

Repo Status Docker image PyPI PyPI License PyPI Python Version Documentation Build Status Coverage pre-commit.ci

What is Data.Rentgen?

Data.Rentgen is a Data Motion Lineage service, compatible with OpenLineage specification.

Currently we support consuming lineage from:

  • Apache Spark

  • Apache Airflow

  • Apache Hive

  • Apache Flink

  • dbt

  • SyncMaster (proprietary integration, part of MWS Data Bridge)

  • StarRocks (proprietary integration, part of MWS Data Engine)

Note: service is under active development, so API can be unstable.

Goals

  • Collect lineage events produced by OpenLineage clients & integrations.

  • Store operation-grained events for better detalization.

  • Provide API for fetching both job/run ↔ dataset lineage and dataset ↔ dataset lineage.

Features

  • Support consuming large amounts of lineage events, use Apache Kafka as event buffer.

  • Store data in tables partitioned by event timestamp, to speed up lineage graph resolution.

  • Lineage graph is build with user-specified time boundaries.

  • Lineage graph can be build with different granularity. e.g. merge all individual Spark commands into Spark applicationId or Spark applicationName.

  • Column-level lineage support.

  • Authentication support.

Non-goals

  • This is not a Data Catalog. DataRentgen doesn’t track dataset schema change, owner and so on. Use Datahub or OpenMetadata instead.

  • Static Data Lineage like view → table is not supported.

Limitations

  • OpenLineage have integrations with Trino, Debezium and some other lineage sources. DataRentgen support may be added later.

  • DataRentgen parses only limited set of OpenLineage facets, and doesn’t store custom facets. This can be changed in future.

Documentation

See https://data-rentgen.readthedocs.io/

Screenshots

Lineage graph

Dataset-level lineage graph

Dataset-level lineage graph

Dataset column-level lineage graph

Dataset column-level lineage graph

Job-level lineage graph

Job-level lineage graph

Run-level lineage graph

Job-level lineage graph

Hierarchy graph

Job hierarchy

Datasets

Datasets list

Runs

Runs list

Spark application

Spark application details

Spark run

Spark run details

Spark command

Spark command details

Hive query

Hive query details

Airflow DagRun

Airflow DagRun details

Airflow TaskInstance

Airflow TaskInstance details

Project links

Data verified by PyPI on Jul 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.7.0)

3 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for dolfinus from gravatar.com dolfinus Avatar for mtsrus from gravatar.com mtsrus Avatar for TiGrib from gravatar.com TiGrib

Credits

Author: MWS Data Bridge

GitHub Statistics

Data verified by PyPI on Jul 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache-2.0

Requires

Python >=3.12

Provides Extra

server consumer http2kafka postgres gssapi seed

Tags

Lineage FastAPI REST FastStream

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

data_rentgen-0.6.0.tar.gz (172.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

data_rentgen-0.6.0-py3-none-any.whl (331.0 kB view details)

Uploaded Python 3

File details

Details for the file data_rentgen-0.6.0.tar.gz.

File metadata

  • Download URL: data_rentgen-0.6.0.tar.gz
  • Upload date:
  • Size: 172.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for data_rentgen-0.6.0.tar.gz
Algorithm Hash digest
SHA256 5fb559ac2cb791fc445f06b96a9db9396311e109a2958a3d32ac46af9701438f
MD5 5e6ececb394e718f5e5e6c6b34e2e26d
BLAKE2b-256 9940fbc555f394aa42d73d4ac7e5df25e3f84b680f96d255e022ed8aa4637260

See more details on using hashes here.

Provenance

The following attestation bundles were made for data_rentgen-0.6.0.tar.gz:

Publisher: release.yml on MTSWebServices/data-rentgen

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file data_rentgen-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: data_rentgen-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 331.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for data_rentgen-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d6fc15377f525ed0f990db83a7127142dec4ef00b9d2f1c96206512423c717be
MD5 ce9fdc261a6e419d9349621ff8ab7e3a
BLAKE2b-256 ccdcc1593076eedcd27a40cecebc4752661dab2f8217a0c08911b673f24b3a3d

See more details on using hashes here.

Provenance

The following attestation bundles were made for data_rentgen-0.6.0-py3-none-any.whl:

Publisher: release.yml on MTSWebServices/data-rentgen

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page