Ingestion Framework for OpenMetadata
Project description
This guide will help you setup the Ingestion framework and connectors
OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites
- Python >= 3.8.x
Install From PyPI
python3 -m pip install --upgrade pip wheel setuptools openmetadata-ingestion
python3 -m spacy download en_core_web_sm
Install Ingestion Connector Dependencies
Click here to go to Ingestion Connector's Documentation
Generate Redshift Data
metadata ingest -c ./pipelines/redshift.json
Generate Redshift Usage Data
metadata ingest -c ./pipelines/redshift_usage.json
Generate Sample Tables
metadata ingest -c ./pipelines/sample_tables.json
Generate Sample Users
metadata ingest -c ./pipelines/sample_users.json
Ingest MySQL data to Metadata APIs
metadata ingest -c ./pipelines/mysql.json
Ingest Bigquery data to Metadata APIs
export GOOGLE_APPLICATION_CREDENTIALS="$PWD/pipelines/creds/bigquery-cred.json"
metadata ingest -c ./pipelines/bigquery.json
Index Metadata into ElasticSearch
Run ElasticSearch docker
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2
Run ingestion connector
metadata ingest -c ./pipelines/metadata_to_es.json
Generated sources
We are using datamodel-codegen
to get some pydantic
classes inside the generated
module from the JSON Schemas defining the API and Entities.
This tool bases the class name on the title
of the JSON Schema (vs. Java POJO, which uses the file name). Note that this convention is important for us, as having a standardized approach in creating the titles helps us create generic code capable of tackling multiple Type Variables.
Changelog
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for openmetadata-ingestion-0.4.11.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3321833bc644c370401b9bbb04a756e03b4b770591bf043f94a5508802c2682e |
|
MD5 | 64c0b6552a81dfd390293336ad1e7647 |
|
BLAKE2b-256 | 4ab22b432b17a71375f4de5c1544c92f104305d46b7ea1b5e29c8edfc344030b |
Hashes for openmetadata_ingestion-0.4.11-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b1d46ef0f84a1f8fb8440301d6e8a0b53c9d8d9d5a79dfde03594dd1ebaa20e |
|
MD5 | 4ffa2947e1ad89fe1a7226daa42f583d |
|
BLAKE2b-256 | 8ace04164748b2fb62d77407d654495a97ff5efb73ef75ced8cf4577454482f6 |