Ingestion Framework for OpenMetadata
Project description
This guide will help you setup the Ingestion framework and connectors
OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites
- Python >= 3.8.x
Install From PyPI
python3 -m pip install --upgrade pip wheel setuptools openmetadata-ingestion
python3 -m spacy download en_core_web_sm
Install Ingestion Connector Dependencies
Click here to go to Ingestion Connector's Documentation
Generate Redshift Data
metadata ingest -c ./pipelines/redshift.json
Generate Redshift Usage Data
metadata ingest -c ./pipelines/redshift_usage.json
Generate Sample Tables
metadata ingest -c ./pipelines/sample_tables.json
Generate Sample Users
metadata ingest -c ./pipelines/sample_users.json
Ingest MySQL data to Metadata APIs
metadata ingest -c ./pipelines/mysql.json
Ingest Bigquery data to Metadata APIs
export GOOGLE_APPLICATION_CREDENTIALS="$PWD/pipelines/creds/bigquery-cred.json"
metadata ingest -c ./pipelines/bigquery.json
Index Metadata into ElasticSearch
Run ElasticSearch docker
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2
Run ingestion connector
metadata ingest -c ./pipelines/metadata_to_es.json
Changelog
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openmetadata-ingestion-0.4.1.tar.gz
(249.0 kB
view hashes)
Built Distribution
Close
Hashes for openmetadata-ingestion-0.4.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3fdb745be7381e35dbe1aa7675b5cd8ae025d26439a8dcb6ab385e1969e8235 |
|
MD5 | 769175b519f1dff224f6ea2e021d96fb |
|
BLAKE2b-256 | a3d7634aa50f689bb8c30edcda73094976440841b12cc521b0bb68f37d336ae4 |
Close
Hashes for openmetadata_ingestion-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b031e16f2c2072c2ee9da4edc0b9f5226ee66f8bf47a629fd3d89296c60ce527 |
|
MD5 | a7aa38d30020dc2cb7ee2b35c47bdbd1 |
|
BLAKE2b-256 | 170f62256b635d472ecb05799a216640fb0aa64dbc24bdfd09cab8e12f46ebab |