A collection of Python-based 'connectors' that extract metadata from various sources to ingest into the Metaphor app.
Project description
Metaphor Connectors
This repository contains a collection of Python-based "connectors" that extract metadata from various sources to ingest into the Metaphor platform.
Installation
This package requires Python 3.9+ installed. You can verify the version on your system by running the following command,
python -V # or python3 on some systems
Once verified, you can install the package using pip,
pip install "metaphor-connectors[all]" # or pip3 on some systems
This will install all the connectors and required dependencies. You can also choose to install only a subset of the dependencies by installing the specific extra, e.g.
pip install "metaphor-connectors[snowflake]"
Similarly, you can also install the package using requirements.txt
or pyproject.toml
.
Docker
We automatically push a docker image to Docker Hub as part of the CI/CD. See this page for more details.
GitHub Action
You can also run the connectors in your CI/CD pipeline using the Metaphor Connectors GitHub Action.
Connectors
Each connector is placed under its own directory under metaphor and extends the metaphor.common.BaseExtractor
class.
Connector Name | Metadata |
---|---|
athena | Schema, description, queries |
azure_data_factory | Lineage, Pipeline |
bigquery | Schema, description, statistics, queries |
bigquery.lineage | Lineage |
bigquery.profile | Data profile |
confluence | Document embeddings |
custom.data_quality | Data quality |
custom.governance | Ownership, tags, description |
custom.lineage | Lineage |
custom.metadata | Custom metadata |
custom.query_attributions | Query attritutions |
datahub | Description, tag, ownership |
dbt | dbt model, test, lineage |
dbt.cloud | dbt model, test, lineage |
fivetran | Lineage, Pipeline |
glue | Schema, description |
great_expectations | Data monitor |
informatica | Lineage, Pipeline |
looker | Looker view, explore, dashboard, lineage |
kafka | Schema, description |
metabase | Dashboard, lineage |
mongodb | Schema, statistics |
monte_carlo | Data monitor |
mssql | Schema |
mysql | Schema, description |
openapi | API, description |
oracle | Schema, description, queries |
notion | Document embeddings |
postgresql | Schema, description, statistics |
postgresql.profile | Data profile |
power_bi | Dashboard, lineage |
quick_sight | Dashboard, lineage |
redshift | Schema, description, statistics, queries |
redshift.profile | Data profile |
s3 | Schema, description |
sharepoint | Document embeddings |
snowflake | Schema, description, statistics, queries |
snowflake.profile | Data profile |
static_web | Document embeddings |
synapse | Schema, queries |
tableau | Dashboard, lineage |
thought_spot | Dashboard, lineage |
trino | Schema, description, queries |
unity_catalog | Schema, description |
unity_catalog.profile | Data profile, statistics |
Development
See Development Environment for more instructions on how to set up your local development environment.
Custom Connectors
See Adding a Custom Connector for instructions and a full example of creating your custom connectors.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for metaphor_connectors-0.14.146.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55fb26aaaa3ae7fa8249e8e594e301ea241fa278aed25afb8a5c3f856b0dd796 |
|
MD5 | 090162015302df6b00e2374d47761444 |
|
BLAKE2b-256 | 3bdead731b7a48a447f16f92d7f08d75df6d8e781e7031fcbef75d19f4a49d41 |
Hashes for metaphor_connectors-0.14.146-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78e6bcb78f25efadec2a46f808ccc69ef12308f50c27d3440329433aeee9e86c |
|
MD5 | af373c2e26c94f453d0505bffac570af |
|
BLAKE2b-256 | f8d1f3b52984e484bf20b6171b8d5c18ca5ac1c0a56def452823de1aa1e73e0b |