Ingestion Framework for MetaMart

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for metamart from gravatar.com metamart

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Digitrans Community License Agreement Version 1.0 This Digitrans Community License Agreement Versio...
  • Author: MetaMart Committers
  • Requires: Python >=3.8
  • Provides-Extra: airflow, all, amundsen, athena, atlas, azure-sso, azuresql, backup, base, bigquery, bigtable, clickhouse, couchbase, dagster, data-insight, databricks, datalake-azure, datalake-gcs, datalake-s3, db2, db2-ibmi, dbt, deltalake, deltalake-spark, deltalake-storage, dev, domo, doris, druid, dynamodb, e2e-test, elasticsearch, extended-testing, glue, great-expectations, greenplum, hive, iceberg, impala, kafka, kafkaconnect, kinesis, looker, mlflow, mongo, mssql, mssql-odbc, mstr, mysql, nifi, openlineage, oracle, pgspider, pii-processor, pinotdb, postgres, powerbi, presto, pymssql, qliksense, quicksight, redash, redpanda, redshift, sagemaker, salesforce, sample-data, sap-hana, sas, singlestore, sklearn, slim, snowflake, superset, tableau, teradata, test, trino, vertica

Project description

— This guide will help you setup the Ingestion framework and connectors —

![Python version 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)

MetaMart Ingestion is a simple framework to build connectors and ingest metadata of various systems through MetaMart APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.8.x

### Docs

Please refer to the documentation here https://docs.meta-mart.org/connectors

<img referrerpolicy=”no-referrer-when-downgrade” src=”https://static.scarf.sh/a.png?x-pxid=c1a30c7c-6dc7-4928-95bf-6ee08ca6aa6a” />

### TopologyRunner

All the Ingestion Workflows run through the TopologyRunner.

The flow is depicted in the images below.

TopologyRunner Standard Flow

![image](../metamart-docs/images/v1.4/features/ingestion/workflows/metadata/multithreading/single-thread-flow.png)

TopologyRunner Multithread Flow

![image](../metamart-docs/images/v1.4/features/ingestion/workflows/metadata/multithreading/multi-thread-flow.png)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for metamart from gravatar.com metamart

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Digitrans Community License Agreement Version 1.0 This Digitrans Community License Agreement Versio...
  • Author: MetaMart Committers
  • Requires: Python >=3.8
  • Provides-Extra: airflow, all, amundsen, athena, atlas, azure-sso, azuresql, backup, base, bigquery, bigtable, clickhouse, couchbase, dagster, data-insight, databricks, datalake-azure, datalake-gcs, datalake-s3, db2, db2-ibmi, dbt, deltalake, deltalake-spark, deltalake-storage, dev, domo, doris, druid, dynamodb, e2e-test, elasticsearch, extended-testing, glue, great-expectations, greenplum, hive, iceberg, impala, kafka, kafkaconnect, kinesis, looker, mlflow, mongo, mssql, mssql-odbc, mstr, mysql, nifi, openlineage, oracle, pgspider, pii-processor, pinotdb, postgres, powerbi, presto, pymssql, qliksense, quicksight, redash, redpanda, redshift, sagemaker, salesforce, sample-data, sap-hana, sas, singlestore, sklearn, slim, snowflake, superset, tableau, teradata, test, trino, vertica

Release history Release notifications | RSS feed

This version

1.6.0.0.dev0 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

metamart_ingestion-1.6.0.0.dev0.tar.gz (1.5 MB view details)

Uploaded Source

Built Distribution

metamart_ingestion-1.6.0.0.dev0-py3-none-any.whl (2.6 MB view details)

Uploaded Python 3

File details

Details for the file metamart_ingestion-1.6.0.0.dev0.tar.gz.

File metadata

File hashes

Hashes for metamart_ingestion-1.6.0.0.dev0.tar.gz
Algorithm Hash digest
SHA256 db252986c9e721ea3ff7ee33e3f43bc57a7fccb7b28d9944223ba296602f0418
MD5 442bb6b2aeab6b6a878793389a8cbd11
BLAKE2b-256 e80a8e11e2b7ca910a9913c8b78953e94659614115002ac9f59d543ce0b364c6

See more details on using hashes here.

File details

Details for the file metamart_ingestion-1.6.0.0.dev0-py3-none-any.whl.

File metadata

File hashes

Hashes for metamart_ingestion-1.6.0.0.dev0-py3-none-any.whl
Algorithm Hash digest
SHA256 d1dc4842fd7f2c57bde627f65de4ae238b865dc81da4b4ca769f4f4bf10306ef
MD5 6b933acaa349995a7e282e9fc67306fc
BLAKE2b-256 f4fecde567e32e5292f3f0faa70dfeba676078d550a1b5b7bb2c3a5964fc6462

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page