Ingestion Framework for OpenMetadata

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for openmetadata from gravatar.com openmetadata

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Collate Community License Agreement
  • Author: OpenMetadata Committers
  • Requires: Python >=3.9
  • Provides-Extra: airflow , all , all-dev-env , amundsen , athena , atlas , azure-sso , azuresql , backup , bigquery , bigtable , cassandra , clickhouse , cockroach , couchbase , dagster , data-insight , databricks , datalake-azure , datalake-gcs , datalake-s3 , db2 , db2-ibmi , dbt , deltalake , deltalake-spark , deltalake-storage , dev , docs , domo , doris , druid , dynamodb , e2e-test , elasticsearch , exasol , glue , great-expectations , great-expectations-1xx , greenplum , hive , iceberg , impala , kafka , kafkaconnect , kinesis , looker , mlflow , mongo , mssql , mssql-odbc , mysql , nifi , openlineage , opensearch , oracle , pandas , pgspider , pii-processor , pinotdb , playwright , postgres , powerbi , presidio-analyzer , presto , pyarrow , pymssql , qliksense , quicksight , redash , redpanda , redshift , sagemaker , salesforce , sample-data , sap-hana , sas , singlestore , sklearn , slim , snowflake , ssrs , starrocks , superset , tableau , teradata , test , test-unit , trino , vertica
Report project as malware

Project description

— This guide will help you setup the Ingestion framework and connectors —

![Python version 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)

OpenMetadata Ingestion is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.9.x

### Docs

Please refer to the documentation here https://docs.open-metadata.org/connectors

<img referrerpolicy=”no-referrer-when-downgrade” src=”https://static.scarf.sh/a.png?x-pxid=c1a30c7c-6dc7-4928-95bf-6ee08ca6aa6a” />

### TopologyRunner

All the Ingestion Workflows run through the TopologyRunner.

The flow is depicted in the images below.

TopologyRunner Standard Flow

![image](https://raw.githubusercontent.com/open-metadata/docs-v1/refs/heads/main/public/images/v1.9/features/ingestion/workflows/metadata/multithreading/single-thread-flow.png)

TopologyRunner Multithread Flow

![image](https://raw.githubusercontent.com/open-metadata/docs-v1/refs/heads/main/public/images/v1.9/features/ingestion/workflows/metadata/multithreading/multi-thread-flow.png)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for openmetadata from gravatar.com openmetadata

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Collate Community License Agreement
  • Author: OpenMetadata Committers
  • Requires: Python >=3.9
  • Provides-Extra: airflow , all , all-dev-env , amundsen , athena , atlas , azure-sso , azuresql , backup , bigquery , bigtable , cassandra , clickhouse , cockroach , couchbase , dagster , data-insight , databricks , datalake-azure , datalake-gcs , datalake-s3 , db2 , db2-ibmi , dbt , deltalake , deltalake-spark , deltalake-storage , dev , docs , domo , doris , druid , dynamodb , e2e-test , elasticsearch , exasol , glue , great-expectations , great-expectations-1xx , greenplum , hive , iceberg , impala , kafka , kafkaconnect , kinesis , looker , mlflow , mongo , mssql , mssql-odbc , mysql , nifi , openlineage , opensearch , oracle , pandas , pgspider , pii-processor , pinotdb , playwright , postgres , powerbi , presidio-analyzer , presto , pyarrow , pymssql , qliksense , quicksight , redash , redpanda , redshift , sagemaker , salesforce , sample-data , sap-hana , sas , singlestore , sklearn , slim , snowflake , ssrs , starrocks , superset , tableau , teradata , test , test-unit , trino , vertica

Release history Release notifications | RSS feed

1.13.0.0rc1 pre-release

This version

1.12.9.0

1.12.8.10

1.12.8.9

1.12.8.8

1.12.8.7

1.12.8.6

1.12.8.4

1.12.8.3

1.12.8.2

1.12.8.1

1.12.8.0

1.12.7.0

1.12.6.4

1.12.6.3

1.12.6.2

1.12.6.1

1.12.6.0

1.12.5.6

1.12.5.5

1.12.5.4

1.12.5.3

1.12.5.2

1.12.5.1

1.12.5.0

1.12.4.1

1.12.4.0

1.12.3.0

1.12.2.0

1.12.1.0

1.12.0.0

1.12.0.0rc1 pre-release

1.11.14.0

1.11.13.0

1.11.12.2

1.11.12.1

1.11.12.0

1.11.11.3

1.11.11.2

1.11.11.1

1.11.11.0

1.11.10.0

1.11.9.2

1.11.9.1

1.11.9.0

1.11.8.1

1.11.8.0

1.11.7.2

1.11.7.1

1.11.7.0

1.11.6.3

1.11.6.2

1.11.6.1

1.11.6.0

1.11.5.1

1.11.5.0

1.11.4.0

1.11.3.0

1.11.2.0

1.11.1.0

1.11.0.0

1.11.0.0rc1 pre-release

1.10.14.7

1.10.14.6

1.10.14.5

1.10.14.4

1.10.14.3

1.10.14.2

1.10.14.1

1.10.14.0

1.10.13.0

1.10.12.0

1.10.11.2

1.10.11.1

1.10.11.0

1.10.10.0

1.10.9.1

1.10.9.0

1.10.8.0

1.10.7.0

1.10.6.0

1.10.5.0

1.10.4.1

1.10.4.0

1.10.3.0

1.10.2.0

1.10.1.0

1.10.0.0

1.9.17.6

1.9.17.5

1.9.17.4

1.9.17.3

1.9.17.2

1.9.17.1

1.9.17.0

1.9.16.2

1.9.16.1

1.9.16.0

1.9.15.0

1.9.14.1

1.9.14.0

1.9.13.1

1.9.13.0 yanked

Reason this release was yanked:

Invalid Release

1.9.12.0

1.9.11.9

1.9.11.8.1

1.9.11.8.0

1.9.11.7.0

1.9.11.6

1.9.11.5

1.9.11.4

1.9.11.3

1.9.11.2

1.9.11.1

1.9.11.0

1.9.10.0

1.9.9.1

1.9.9.0

1.9.8.0

1.9.7.1

1.9.7.0

1.9.6.0

1.9.5.0

1.9.4.0

1.9.3.0

1.9.2.1

1.9.2.0

1.9.1.0

1.9.0.0

1.8.12.0

1.8.11.0

1.8.10.0

1.8.9.0

1.8.8.1

1.8.8.0

1.8.7.5

1.8.7.4

1.8.7.3

1.8.7.2

1.8.7.1

1.8.7.0

1.8.6.1

1.8.6.0

1.8.5.0

1.8.4.0

1.8.3.1

1.8.3.0

1.8.2.0

1.8.1.0

1.8.0.0

1.7.7.4

1.7.7.3

1.7.7.2

1.7.7.1

1.7.7.0

1.7.6.0

1.7.5.4

1.7.5.3

1.7.5.2

1.7.5.1

1.7.5.0

1.7.4.2

1.7.4.1

1.7.4.0

1.7.3.0

1.7.2.1

1.7.2.0

1.7.1.5

1.7.1.4

1.7.1.3

1.7.1.2

1.7.1.1

1.7.1.0

1.7.0.1

1.7.0.0

1.7.0.0rc2 pre-release

1.7.0.0rc1 pre-release

1.6.13.2

1.6.13.1

1.6.13.0

1.6.12.0

1.6.11.1

1.6.11.0

1.6.10.0

1.6.9.0

1.6.8.0

1.6.7.3

1.6.7.2

1.6.7.1

1.6.7.0

1.6.6.9

1.6.6.8

1.6.6.7

1.6.6.6

1.6.6.5

1.6.6.4

1.6.6.3 yanked

1.6.6.2

1.6.6.1

1.6.6.0

1.6.5.2

1.6.5.1

1.6.5.0

1.6.4.0

1.6.3.3

1.6.3.2

1.6.3.1

1.6.3.0

1.6.2.2

1.6.2.1

1.6.2.0

1.6.1.0

1.6.0.0

1.6.0.0rc3 pre-release

1.6.0.0rc2 pre-release

1.6.0.0rc1 pre-release

1.5.15.2

1.5.15.1

1.5.15.0

1.5.14.1

1.5.14.0

1.5.13.2

1.5.13.1

1.5.13.0

1.5.12.1

1.5.12.0

1.5.11.0

1.5.10.0

1.5.9.0

1.5.8.0

1.5.7.1

1.5.7.0

1.5.6.0

1.5.5.0

1.5.4.1

1.5.4.0

1.5.3.1

1.5.3.0

1.5.2.0

1.5.1.0

1.5.0.0

1.5.0.0rc2 pre-release

1.5.0.0rc1 pre-release

1.4.8.1

1.4.8.0

1.4.7.2

1.4.7.1

1.4.7.0

1.4.6.0

1.4.6.0rc1 pre-release

1.4.5.0

1.4.5.0rc1 pre-release

1.4.4.2

1.4.4.1

1.4.4.0

1.4.4.0rc1 pre-release

1.4.3.3

1.4.3.2

1.4.3.1

1.4.3.0

1.4.2.0

1.4.1.0

1.4.0.1

1.4.0.0

1.4.0.0rc4 pre-release

1.4.0.0rc3 pre-release

1.4.0.0rc2 pre-release

1.4.0.0rc1 pre-release

1.3.4.0

1.3.3.0

1.3.3.0rc2 pre-release

1.3.3.0rc1 pre-release

1.3.2.0

1.3.2.0rc3 pre-release

1.3.2.0rc2 pre-release

1.3.2.0rc1 pre-release

1.3.1.3

1.3.1.2

1.3.1.1

1.3.1.0

1.3.0.1

1.3.0.0

1.2.5.3

1.2.5.2

1.2.5.1

1.2.5.0

1.2.4.7

1.2.4.6

1.2.4.5

1.2.4.4

1.2.4.3

1.2.4.2

1.2.4.1

1.2.4.0

1.2.3.3

1.2.3.2

1.2.3.1

1.2.3.0

1.2.2.2

1.2.2.1

1.2.2.0

1.2.1.1

1.2.1.0

1.2.0.1

1.2.0.0

1.1.7.5

1.1.7.4

1.1.7.3

1.1.7.2

1.1.7.1

1.1.7.0

1.1.6.1

1.1.6.0

1.1.5.0

1.1.4.0

1.1.3.0

1.1.2.0

1.1.1.1

1.1.1.0

1.1.0.5

1.1.0.4

1.1.0.3

1.1.0.2

1.1.0.1

1.1.0.0

1.0.5.1

1.0.5.0

1.0.4.4

1.0.4.3

1.0.4.2

1.0.4.1

1.0.4.0

1.0.3.0

1.0.2.0

1.0.1.3

1.0.1.2 yanked

Reason this release was yanked:

openmetadata-managed-apis had 504 gateway timeout while publishing. Yanking this version as well as we maintain same versions for both packages.

1.0.1.1

1.0.1.0

1.0.0.2

1.0.0.1

1.0.0.0

0.13.4.0

0.13.3.1

0.13.3.0

0.13.2.8

0.13.2.7

0.13.2.6

0.13.2.5

0.13.2.4

0.13.2.3

0.13.2.2

0.13.2.1

0.13.2.0

0.13.2.0b0 pre-release

0.13.1.9

0.13.1.8

0.13.1.7

0.13.1.6

0.13.1.5

0.13.1.4

0.13.1.3

0.13.1.2

0.13.1.1

0.13.1.0

0.13.0.3

0.13.0.1

0.13.0.0

0.12.3.2

0.12.3.1

0.12.3.0

0.12.2.4

0.12.2.3

0.12.2.2

0.12.2.1

0.12.2.0

0.12.1.3

0.12.1.2

0.12.1.1

0.12.1.0

0.12.0.3

0.12.0.2

0.12.0.1

0.12.0.0

0.11.5.3

0.11.5.2

0.11.5.1

0.11.5.0

0.11.4.4

0.11.4.3

0.11.4.2

0.11.4.1

0.11.4.0

0.11.3.1

0.11.3.0

0.11.2.0

0.11.1.1

0.11.1.0

0.11.0.0

0.10.4.1

0.10.4.0

0.10.3.2

0.10.3.1

0.10.3.0

0.10.2.0

0.10.1.1

0.10.1

0.10.0.1

0.10.0

0.9.2

0.9.1

0.9.0

0.8.16

0.8.15

0.8.14

0.8.13

0.8.12

0.8.11

0.8.10

0.8.8

0.8.7

0.8.6

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.4

0.7.3

0.7.2

0.7.1

0.5.0

0.4.11

0.4.10

0.4.9

0.4.8

0.4.7

0.4.5

0.4.4

0.4.3

0.4.2.1

0.4.2

0.4.1

0.4.0

0.3.2

0.3.0

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

openmetadata_ingestion-1.12.9.0.tar.gz (20.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

openmetadata_ingestion-1.12.9.0-py3-none-any.whl (22.2 MB view details)

Uploaded Python 3

File details

Details for the file openmetadata_ingestion-1.12.9.0.tar.gz.

File metadata

File hashes

Hashes for openmetadata_ingestion-1.12.9.0.tar.gz
Algorithm Hash digest
SHA256 f34914798a984adf10fe977903a0b139213a66c23b981eac50100eeee82db8b9
MD5 de521e14ab5c71eb5cb065116b23164e
BLAKE2b-256 2b5a895ca0d869f6b1a25b3ceb6cfb8b1e57560e8e3740be5fa0a3b5b627b555

See more details on using hashes here.

File details

Details for the file openmetadata_ingestion-1.12.9.0-py3-none-any.whl.

File metadata

File hashes

Hashes for openmetadata_ingestion-1.12.9.0-py3-none-any.whl
Algorithm Hash digest
SHA256 1da75385012cb00d9d3202eca4d7ae9c79093428a45dfc75deee9d364bccfc81
MD5 782253b17e2a7da97f57683fed6c0555
BLAKE2b-256 ee7034cf6ea5ad48bfabb5213af885e5ce66ca6d463ad52f2b88e2861222658d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page