Skip to main content

A collection of Python-based 'connectors' that extract metadata from various sources to ingest into the Metaphor app.

Project description

Metaphor Connectors

Codecov CodeQL PyPI Version Python version 3.9+ PyPI Downloads Docker Pulls License

This repository contains a collection of Python-based "connectors" that extract metadata from various sources to ingest into the Metaphor platform.

Installation

This package requires Python 3.9+ installed. You can verify the version on your system by running the following command,

python -V  # or python3 on some systems

Once verified, you can install the package using pip,

pip install "metaphor-connectors[all]"  # or pip3 on some systems

This will install all the connectors and required dependencies. You can also choose to install only a subset of the dependencies by installing the specific extra, e.g.

pip install "metaphor-connectors[snowflake]"

Similarly, you can also install the package using requirements.txt or pyproject.toml.

Docker

We automatically push a docker image to Docker Hub as part of the CI/CD. See this page for more details.

GitHub Action

You can also run the connectors in your CI/CD pipeline using the Metaphor Connectors GitHub Action.

Connectors

Each connector is placed under its own directory under metaphor and extends the metaphor.common.BaseExtractor class.

Connector Name Metadata
athena Schema, description, queries
azure_data_factory Lineage, Pipeline
bigquery Schema, description, statistics, queries
bigquery.lineage Lineage
bigquery.profile Data profile
confluence Document embeddings
custom.data_quality Data quality
custom.governance Ownership, tags, description
custom.lineage Lineage
custom.metadata Custom metadata
custom.query_attributions Query attritutions
datahub Description, tag, ownership
dbt dbt model, test, lineage
dbt.cloud dbt model, test, lineage
fivetran Lineage, Pipeline
glue Schema, description
great_expectations Data monitor
informatica Lineage, Pipeline
looker Looker view, explore, dashboard, lineage
kafka Schema, description
metabase Dashboard, lineage
mongodb Schema, statistics
monte_carlo Data monitor
mssql Schema
mysql Schema, description
openapi API, description
oracle Schema, description, queries
notion Document embeddings
postgresql Schema, description, statistics
postgresql.profile Data profile
power_bi Dashboard, lineage
quick_sight Dashboard, lineage
redshift Schema, description, statistics, queries
redshift.profile Data profile
s3 Schema, description
sharepoint Document embeddings
snowflake Schema, description, statistics, queries
snowflake.profile Data profile
static_web Document embeddings
synapse Schema, queries
tableau Dashboard, lineage
thought_spot Dashboard, lineage
trino Schema, description, queries
unity_catalog Schema, description
unity_catalog.profile Data profile, statistics

Development

See Development Environment for more instructions on how to set up your local development environment.

Custom Connectors

See Adding a Custom Connector for instructions and a full example of creating your custom connectors.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

metaphor_connectors-0.14.159.tar.gz (346.7 kB view details)

Uploaded Source

Built Distribution

metaphor_connectors-0.14.159-py3-none-any.whl (482.5 kB view details)

Uploaded Python 3

File details

Details for the file metaphor_connectors-0.14.159.tar.gz.

File metadata

  • Download URL: metaphor_connectors-0.14.159.tar.gz
  • Upload date:
  • Size: 346.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.9.20 Linux/6.5.0-1025-azure

File hashes

Hashes for metaphor_connectors-0.14.159.tar.gz
Algorithm Hash digest
SHA256 e0233f6e0e05f0662769af014d08009a71bb41a0f9367933b3ec8527dde1b0be
MD5 435d88707a31e9ff2ce1e0f1bc03c344
BLAKE2b-256 a5fecd153aaa1195566d5604938f1661c2905201dc07739c991f1476c61909c3

See more details on using hashes here.

File details

Details for the file metaphor_connectors-0.14.159-py3-none-any.whl.

File metadata

File hashes

Hashes for metaphor_connectors-0.14.159-py3-none-any.whl
Algorithm Hash digest
SHA256 d896d3af92c874bff05a33ec86723288653591b0eb2537984385d41b2bddae78
MD5 997e29c3b61f6ad898cbcbc61c415a87
BLAKE2b-256 37d2076ba90d9b1e13cf61d8316a224bb5640c97741474564a1b7b7d9dd92450

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page