A CLI to work with DataHub metadata
Project description
Introduction to Metadata Ingestion
:::tip Find Integration Source Please see our Integrations page to browse our ingestion sources and filter on their features. :::
Integration Methods
DataHub offers three methods for data ingestion:
- UI Ingestion : Easily configure and execute a metadata ingestion pipeline through the UI.
- CLI Ingestion guide : Configure the ingestion pipeline using YAML and execute by it through CLI.
- SDK-based ingestion : Use Python Emitter or Java emitter to programmatically control the ingestion pipelines.
Types of Integration
Integration can be divided into two concepts based on the method:
Push-based Integration
Push-based integrations allow you to emit metadata directly from your data systems when metadata changes. Examples of push-based integrations include Airflow, Spark, Great Expectations and Protobuf Schemas. This allows you to get low-latency metadata integration from the "active" agents in your data ecosystem.
Pull-based Integration
Pull-based integrations allow you to "crawl" or "ingest" metadata from the data systems by connecting to them and extracting metadata in a batch or incremental-batch manner. Examples of pull-based integrations include BigQuery, Snowflake, Looker, Tableau and many others.
Core Concepts
The following are the core concepts related to ingestion:
- Sources: Data systems from which extract metadata. (e.g. BigQuery, MySQL)
- Sinks: Destination for metadata (e.g. File, DataHub)
- Recipe: The main configuration for ingestion in the form or .yaml file
For more advanced guides, please refer to the following:
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acryl_datahub-1.5.0.1.tar.gz.
File metadata
- Download URL: acryl_datahub-1.5.0.1.tar.gz
- Upload date:
- Size: 3.0 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3f0c81210f827b314cc1ad957cc2fe027a0f590e6dab0c8fbc54d5c637348624
|
|
| MD5 |
8d0f8b5e4cff5075c982f137857b7444
|
|
| BLAKE2b-256 |
778650498c0d31ed8a9d21961f9e861af38710e69221e959dc165cdb2c3dfe88
|
File details
Details for the file acryl_datahub-1.5.0.1-py3-none-any.whl.
File metadata
- Download URL: acryl_datahub-1.5.0.1-py3-none-any.whl
- Upload date:
- Size: 3.6 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
04a06574aa8df22cf9b8e6b85d4c569d32a0796b24caa87a23a9f81f87914aa4
|
|
| MD5 |
1a85ea16571652018f122b2feb5845f9
|
|
| BLAKE2b-256 |
8579701d1f9d6bf20ecdc89f832f53ee8ab500f330920edc8bc3009dda86ce9a
|
