Extract table lineage from Postgres views
Project description
Datahub Postgres View Lineage
A ingestion source to generate lineage for views in a Postgres database.
Quick Start
First install Poetry and task and initialize the project
task setup
Now, start a database
task start wait sample-view
Now run the ingestion to the console
task run
When it is successful, the output should include
Source (datahub_postgres_lineage.ingestion.PostgresLineageSource) report:
{'events_produced': '1',
'events_produced_per_sec': '26',
'event_ids': ['urn:li:dataset:(urn:li:dataPlatform:postgres,cool_db.public.emails,PROD)-upstreamLineage'],
'warnings': {},
'failures': {},
'filtered': ['public.names'],
'start_time': '2022-12-20 16:09:46.105046 (now).',
'running_time': '0.04 seconds'}
Configuration
Key | Description | Default |
---|---|---|
username |
The username to connect to the database | '' |
password |
The password to connect to the database | '' |
host_port |
The host and port to connect to the database | '' |
database |
The database to connect to | '' |
database_alias |
Alias to apply to database when ingesting. | '' |
sqlalchemy_uri |
SQLAlchemy URI to connect to the database | '' |
scheme |
The SQLAlchemy scheme to use | postgressql+psycopg2 |
schema_pattern |
||
schema_pattern.allow |
Regexp pattern to match schemas to include | .* |
schema_pattern.deny |
Regexp pattern to match schemas to exclude, 'information_schema' and 'pg_catalog' are already excluded | '' |
view_pattern |
||
view_pattern.allow |
Regexp pattern to match view names to include | .* |
view_pattern.deny |
Regexp pattern to match view names to exclude | '' |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for datahub_postgres_lineage-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 065a401cc7712ea65fced253e490a9f8f96a6c55766bad58cbc4c60c740a2e82 |
|
MD5 | 7f4241d1c56bc828ca9ff4ba804c3a07 |
|
BLAKE2b-256 | d144eb3e57dfc3c48cbaaa4327d5cde9a3bafdfc01547e4ab5b87753475d87df |
Close
Hashes for datahub_postgres_lineage-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1095ec38255c037e928274f3c37be973c32ceb54b97b49e82c3418876197155 |
|
MD5 | 265dea04942d4e019ceef994fe1dffaf |
|
BLAKE2b-256 | 2aa050b010253d5b7dd57f7c55578b21d8f949f10f90280ab6810f9c81a81822 |