Presenting data lineage based on your data warehouse query history
Project description
Elementary was built out of the need to effortlessly and immediately gain visibility into the data stack, starting with tracing the actual upstream & downstream dependencies in the data warehouse, without any implementation efforts, security risks or compromises on accuracy.
Data lineage made simple, reliable, and automated.
(Some features are still under construction :construction:)
:star: If you like what we are building, support us with a :star:
Quick start
pip install elementary-lineage
# The tool is named edl (Elementary Data Lineage),
# run it to validate the installation:
edl --help
We need a connection file in a simple YAML called profiles.yml
. Here is a template to create a Snowflake / BigQuery one.
For further instructions go to our quickstart page.
If you use dbt, you can start right away by running this command with the path to your profiles.yml
and the relevant profile name:
edl -d ~/profiles_dir -p <profile_name>
Documentation
Our full documentation is available here.
Features
- Lineage visualization: Visual map of data flow and dependencies in the data warehouse.
- Dataset status: Present data about freshness and volume on the lineage graph.
- Accuracy: Reflects the actual state in the DWH based on logs.
- Plug-and-play: No need for code changes.
- Graph filters: Filter the graph by dataset, dates, direction, and depth.
Coming soon:
- Dataset monitoring: Continuously collect and monitor datasets' reliability metrics.
- Lineage history: Store data about lineage versions and changes.
- Column level lineage: Add column-level granularity.
- Full lineage: Integrate with downstream and upstream tools to create a full graph.
Our plan is to eventually build a data observability platform that is open, transparent and powered by the community. A solution that data teams could easily integrate into their workflows, detect data incidents and prevent them from even happening in the first place.
Community & Support
For additional information and help, you can use one of these channels:
- Slack (Live chat with the team, support, discussions, etc.)
- GitHub issues (Bug reports, feature requests)
- Twitter (Updates on new releases and stuff)
Integrations
- Snowflake
- BigQuery
- Redshift
Ask us for integrations on Slack or as a GitHub issue.
License
Elementary lineage is licensed under Apache License 2.0. See the LICENSE file for licensing information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for elementary-lineage-0.0.12.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cbea7af6acaf742a08f9d9b0d3ca13ea9106fe6b0b96ad905042bce564ebcd7 |
|
MD5 | f8505386a390add323fa659e0b7a1a95 |
|
BLAKE2b-256 | e82f8b83794f6f670a8b6ea8005881a720dce61d4ef5b20cc180735f22f7b358 |
Hashes for elementary_lineage-0.0.12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2ecc88c0d627b6a8d41eec244bc0ef478869e93ba1e1836c0d0816d6deb8e79 |
|
MD5 | d717623ea2b09fce5293baddcdc70884 |
|
BLAKE2b-256 | 6712aa8e125bc302601308fe036e0c294530afd3d3b8b9fa648d61f4b5ad288c |