Intro to Metadata Ingestion
This module hosts an extensible Python-based metadata ingestion system for DataHub. This supports sending data to DataHub using Kafka or through the REST API. It can be used through our CLI tool, with an orchestrator like Airflow, or as a library.
Getting Started
Prerequisites
Before running any metadata ingestion job, you should make sure that DataHub backend services are all running. If you are trying this out locally check out the CLI to install the CLI and understand the options available in the CLI. You can reference the CLI usage guide given there as you go through this page.
Recipes
A recipe is a configuration file that tells our ingestion scripts where to pull data from (source) and where to put it (sink). Here's a simple example that pulls metadata from MSSQL (source) and puts it into datahub rest (sink).
Note that one recipe file can only have 1 source and 1 sink. If you want multiple sources then you will need multiple recipe files.
# A sample recipe that pulls metadata from MSSQL and puts it into DataHub
# using the Rest API.
source:
type: mssql
config:
username: sa
password: ${MSSQL_PASSWORD}
database: DemoData
transformers:
- type: "fully-qualified-class-name-of-transformer"
config:
some_property: "some.value"
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
A number of recipes are included in the examples/recipes directory. For full info and context on each source and sink, see the pages described in the table of plugins.
Handling sensitive information in recipes
We automatically expand environment variables in the config (e.g. ${MSSQL_PASSWORD}),
similar to variable substitution in GNU bash or in docker-compose files. For details, see
https://docs.docker.com/compose/compose-file/compose-file-v2/#variable-substitution. This environment variable substitution should be used to mask sensitive information in recipe files. As long as you can get env variables securely to the ingestion process there would not be any need to store sensitive information in recipes.
Basic Usage of CLI for ingestion
pip install 'acryl-datahub[datahub-rest]' # install the required plugin
datahub ingest -c ./examples/recipes/mssql_to_datahub.yml
The --dry-run option of the ingest command performs all of the ingestion steps, except writing to the sink. This is useful to ensure that the
ingestion recipe is producing the desired workunits before ingesting them into datahub.
# Dry run
datahub ingest -c ./examples/recipes/example_to_datahub_rest.yml --dry-run
# Short-form
datahub ingest -c ./examples/recipes/example_to_datahub_rest.yml -n
The --preview option of the ingest command performs all of the ingestion steps, but limits the processing to only the first 10 workunits produced by the source.
This option helps with quick end-to-end smoke testing of the ingestion recipe.
# Preview
datahub ingest -c ./examples/recipes/example_to_datahub_rest.yml --preview
# Preview with dry-run
datahub ingest -c ./examples/recipes/example_to_datahub_rest.yml -n --preview
Transformations
If you'd like to modify data before it reaches the ingestion sinks – for instance, adding additional owners or tags – you can use a transformer to write your own module and integrate it with DataHub.
Check out the transformers guide for more info!
Using as a library
In some cases, you might want to construct Metadata events directly and use programmatic ways to emit that metadata to DataHub. In this case, take a look at the Python emitter and the Java emitter libraries which can be called from your own code.
Programmatic Pipeline
In some cases, you might want to configure and run a pipeline entirely from within your custom python script. Here is an example of how to do it.
- programmatic_pipeline.py - a basic mysql to REST programmatic pipeline.
Developing
See the guides on developing, adding a source and using transformers.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acryl-datahub-0.8.28.0.tar.gz.
File metadata
- Download URL: acryl-datahub-0.8.28.0.tar.gz
- Upload date:
- Size: 511.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
862145594ca22c58cb41313920c8e27f149f10563469816eba213c2601c0167b
|
|
| MD5 |
9f69ec50851b8ffb4c129d86d917c565
|
|
| BLAKE2b-256 |
d329b1d912b7d37dbd882fb0b5ba8efc0aba8691693d42a556def49af43c7c10
|
File details
Details for the file acryl_datahub-0.8.28.0-py3-none-any.whl.
File metadata
- Download URL: acryl_datahub-0.8.28.0-py3-none-any.whl
- Upload date:
- Size: 664.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d3a26485f0fa09e459fcaf44a011844d02adb5e030c1fc3101ec551a3dc61fa2
|
|
| MD5 |
1e5eb9b02d1e3885647499643f169a77
|
|
| BLAKE2b-256 |
08670c41388d456b10b4a9a153510fea8a1a51758641661210f18bce1a5bff62
|
File details
Details for the file acryl_datahub-0.8.28-py3-none-any.whl.
File metadata
- Download URL: acryl_datahub-0.8.28-py3-none-any.whl
- Upload date:
- Size: 665.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fdb4ff77bb597b72de0275405eb03a89bd4e49cd631b3151b8a9809f48a60db0
|
|
| MD5 |
31730ed2c31b534f6c2fa0c2aa8d0bf5
|
|
| BLAKE2b-256 |
481a3aa1baf3c18170b1f77aeb3b2e5896235f6fe638a21ae21c64aeeca59d9f
|
