DataHub SDK and CLI
DataHub's ingestion framework and CLI — pull metadata from 50+ data sources into your catalog, or push it programmatically from your own pipelines and applications.
Pull-based integrations crawl your data systems on a schedule: Snowflake, BigQuery, dbt, Looker, Airflow, and many more.
Push-based integrations let you emit metadata directly from code as it happens: Python SDK, Java SDK, Spark, Great Expectations, and others.
What you can do
- Pull metadata from databases, warehouses, BI tools, and orchestrators using 50+ ready-made connectors
- Push metadata programmatically using the Python or Java SDK
- Transform and filter metadata in transit using built-in transformers
- Schedule and manage ingestion pipelines via CLI or the DataHub UI
- Automate lineage, ownership, tags, and documentation across your data assets
Supported sources
Snowflake · BigQuery · Redshift · dbt · Databricks · Looker · Tableau · Power BI · Airflow · Spark · Kafka · PostgreSQL · MySQL · Hive · Glue · S3 · Iceberg · Unity Catalog · Sigma · Mode · Superset · Metabase · and many more
Installation
pip install acryl-datahub
datahub version
Quickstart
Pull metadata from a source (recipe)
# snowflake_recipe.yml
source:
type: snowflake
config:
account_id: my_account
username: my_user
password: my_password
role: DATAHUB_ROLE
warehouse: COMPUTE_WH
sink:
type: datahub-rest
config:
server: http://localhost:8080
datahub ingest -c snowflake_recipe.yml
Emit metadata from code (SDK)
from datahub.sdk import DataHubClient, Dataset
client = DataHubClient.from_env()
dataset = Dataset(platform="snowflake", name="mydb.schema.table")
dataset.set_description("My table description")
dataset.set_owners(["urn:li:corpuser:jane"])
client.entities.upsert(dataset)
Ingest via the DataHub UI
No CLI required — configure and run ingestion directly from the DataHub UI under Ingestion → Create new source.
Ingestion methods
| Method | Best for |
|---|---|
| CLI + YAML recipe | Scheduled batch ingestion, CI/CD pipelines |
| Python SDK | Programmatic or event-driven metadata emission |
| Java SDK | JVM-based integrations (Spark, Flink, etc.) |
| UI Ingestion | One-click setup, no code required |
Key CLI commands
datahub init # Connect to a DataHub instance (interactive)
datahub init --username datahub --password datahub # Quickstart with local defaults
datahub ingest -c recipe.yml # Run an ingestion pipeline
datahub search "my table" # Search for entities by keyword
datahub search "*" --filter platform=snowflake --filter entity_type=dataset
datahub graphql --list-operations # Explore all available GraphQL operations
datahub graphql --query "{ me { corpUser { urn } } }" # Run a GraphQL query
datahub get --urn <urn> # Fetch metadata for an entity
datahub delete --urn <urn> # Delete a metadata entity
datahub timeline --urn <urn> # View metadata change history
Links
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acryl_datahub-1.7.0.4.tar.gz.
File metadata
- Download URL: acryl_datahub-1.7.0.4.tar.gz
- Upload date:
- Size: 4.3 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
563278d9f9f2959e832a77d953e95185e5610882a7e49424c1f049569dd0a9ca
|
|
| MD5 |
1f72dc27592895ea4b752a2c300a4b73
|
|
| BLAKE2b-256 |
0d1a4c864ced467d65acafc97a37c11383b2fadf9865402edce6f6177f9c3a3e
|
File details
Details for the file acryl_datahub-1.7.0.4-py3-none-any.whl.
File metadata
- Download URL: acryl_datahub-1.7.0.4-py3-none-any.whl
- Upload date:
- Size: 5.1 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6c93888fa451a1f872f91b590bb01449d8c0bf43d9cad87c51f1b46fb8b2bb23
|
|
| MD5 |
f359615cfd005f73cf915d1dafe2f972
|
|
| BLAKE2b-256 |
8002e13ece10460667d2fd37946785e8bb5d112efddd4a3ef929ec39c03646ff
|
