Extract table lineage from Postgres views
Project description
Datahub Postgres View Lineage
A ingestion source to generate lineage for views in a Postgres database.
Quick Start
First install Poetry and task and initialize the project
task setup
Now, start a database
task start wait sample-view
Now run the ingestion to the console
task run
When it is successful, the output should include
Source (datahub_postgres_lineage.ingestion.PostgresLineageSource) report:
{'events_produced': '1',
'events_produced_per_sec': '26',
'event_ids': ['urn:li:dataset:(urn:li:dataPlatform:postgres,cool_db.public.emails,PROD)-upstreamLineage'],
'warnings': {},
'failures': {},
'filtered': ['public.names'],
'start_time': '2022-12-20 16:09:46.105046 (now).',
'running_time': '0.04 seconds'}
Configuration
Key | Description | Default |
---|---|---|
username |
The username to connect to the database | '' |
password |
The password to connect to the database | '' |
host_port |
The host and port to connect to the database | '' |
database |
The database to connect to | '' |
database_alias |
Alias to apply to database when ingesting. | '' |
sqlalchemy_uri |
SQLAlchemy URI to connect to the database | '' |
scheme |
The SQLAlchemy scheme to use | postgressql+psycopg2 |
schema_pattern |
||
schema_pattern.allow |
Regexp pattern to match schemas to include | .* |
schema_pattern.deny |
Regexp pattern to match schemas to exclude, 'information_schema' and 'pg_catalog' are already excluded | '' |
view_pattern |
||
view_pattern.allow |
Regexp pattern to match view names to include | .* |
view_pattern.deny |
Regexp pattern to match view names to exclude | '' |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for datahub_postgres_lineage-0.1.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5ee5b31db630d0502a1e3a166eb126306e321b3fc37272dccb1aba485260564 |
|
MD5 | ab320ba3587db56a63f4d6aeedc01123 |
|
BLAKE2b-256 | 9c1a229cf3c1875fc5c769dae087c0c0d0760405770e702826a8223a5a7b888c |
Close
Hashes for datahub_postgres_lineage-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 204828f768b4394ef16149916a70ddd08dbc852e6b69cc640e908c90d7b6f1b1 |
|
MD5 | 4ecbeea2c6ed2ca178c8d8bc94398f49 |
|
BLAKE2b-256 | 6b4ab04026375ab3ceee56a285cdc485126682c63c5d14c52b0e8f0f283a0f1e |