Open Source Data Lineage Tool For AWS and GCP
Project description
Tokern Lineage Engine
Tokern Lineage Engine is fast and easy to use application to collect, visualize and analyze column-level data lineage in databases, data warehouses and data lakes in AWS and GCP.
Tokern Lineage helps you browse column-level data lineage
- visually using kedro-viz
- analyze lineage graphs programmatically using the powerful networkx graph library
Resources
- Demo of Tokern Lineage App
-
Checkout an example data lineage notebook.
-
Check out the post on using data lineage for cost control for an example of how data lineage can be used in production.
Quick Start
Install a demo of using Docker and Docker Compose
Download the docker-compose file from Github repository.
# in a new directory run
wget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose-demodb/docker-compose.yml
# or run
curl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose-demodb/docker-compose.yml -o docker-compose.yml
Run docker-compose
docker-compose up -d
Check that the containers are running.
docker ps
CONTAINER ID IMAGE CREATED STATUS PORTS NAMES
3f4e77845b81 tokern/data-lineage-viz:latest ... 4 hours ago Up 4 hours 0.0.0.0:8000->80/tcp tokern-data-lineage-visualizer
1e1ce4efd792 tokern/data-lineage:latest ... 5 days ago Up 5 days tokern-data-lineage
38be15bedd39 tokern/demodb:latest ... 2 weeks ago Up 2 weeks tokern-demodb
Try out Tokern Lineage App
Head to http://localhost:8000/
to open the Tokern Lineage app
Supported Technologies
- Postgres
- AWS Redshift
- Snowflake
Coming Soon
- MySQL
- SparkSQL
- Presto
Documentation
For advanced usage, please refer to data-lineage documentation
Survey
Please take this survey if you are a user or considering using data-lineage. Responses will help us prioritize features better.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for data_lineage-0.7.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b161be01f9f1cf0eba7e7edb5b227b37c900cb070eb62752337c4317ee0c714d |
|
MD5 | 97e4f27abf5a62458cece3e7cb806d97 |
|
BLAKE2b-256 | 829a347cbd1944c3fc2c9bb38049e55bb9118dcf18103fef18a059cf03a4ea2f |