DataHub SDK and CLI

DataHub's ingestion framework and CLI — pull metadata from 50+ data sources into your catalog, or push it programmatically from your own pipelines and applications.

Pull-based integrations crawl your data systems on a schedule: Snowflake, BigQuery, dbt, Looker, Airflow, and many more.

Push-based integrations let you emit metadata directly from code as it happens: Python SDK, Java SDK, Spark, Great Expectations, and others.

What you can do

Pull metadata from databases, warehouses, BI tools, and orchestrators using 50+ ready-made connectors
Push metadata programmatically using the Python or Java SDK
Transform and filter metadata in transit using built-in transformers
Schedule and manage ingestion pipelines via CLI or the DataHub UI
Automate lineage, ownership, tags, and documentation across your data assets

Supported sources

Snowflake · BigQuery · Redshift · dbt · Databricks · Looker · Tableau · Power BI · Airflow · Spark · Kafka · PostgreSQL · MySQL · Hive · Glue · S3 · Iceberg · Unity Catalog · Sigma · Mode · Superset · Metabase · and many more

Installation

pip install acryl-datahub
datahub version

Quickstart

Pull metadata from a source (recipe)

# snowflake_recipe.yml
source:
  type: snowflake
  config:
    account_id: my_account
    username: my_user
    password: my_password
    role: DATAHUB_ROLE
    warehouse: COMPUTE_WH

sink:
  type: datahub-rest
  config:
    server: http://localhost:8080

datahub ingest -c snowflake_recipe.yml

Emit metadata from code (SDK)

from datahub.sdk import DataHubClient, Dataset

client = DataHubClient.from_env()

dataset = Dataset(platform="snowflake", name="mydb.schema.table")
dataset.set_description("My table description")
dataset.set_owners(["urn:li:corpuser:jane"])

client.entities.upsert(dataset)

Ingest via the DataHub UI

No CLI required — configure and run ingestion directly from the DataHub UI under Ingestion → Create new source.

Ingestion methods

Method	Best for
CLI + YAML recipe	Scheduled batch ingestion, CI/CD pipelines
Python SDK	Programmatic or event-driven metadata emission
Java SDK	JVM-based integrations (Spark, Flink, etc.)
UI Ingestion	One-click setup, no code required

Key CLI commands

datahub init                          # Connect to a DataHub instance (interactive)
datahub init --username datahub --password datahub  # Quickstart with local defaults
datahub ingest -c recipe.yml          # Run an ingestion pipeline
datahub search "my table"             # Search for entities by keyword
datahub search "*" --filter platform=snowflake --filter entity_type=dataset
datahub graphql --list-operations     # Explore all available GraphQL operations
datahub graphql --query "{ me { corpUser { urn } } }"  # Run a GraphQL query
datahub get --urn <urn>               # Fetch metadata for an entity
datahub delete --urn <urn>            # Delete a metadata entity
datahub timeline --urn <urn>          # View metadata change history

Links

base abs abs-slim aerospike airbyte athena aws-secret-manager azure-ad azure-auth azure-data-factory bigid bigquery bigquery-queries bigquery-slim cassandra circuit-breaker clickhouse clickhouse-usage cockroachdb confluence cube databricks datahub datahub-business-glossary datahub-debug datahub-documents datahub-gc datahub-kafka datahub-lineage-file datahub-lite datahub-pg-queue datahub-rest dataplex db2 dbt dbt-cloud debug-recording delta-lake dlt doris dremio druid dynamodb elasticsearch excel fabric-data-factory fabric-onelake feast fivetran flink gcp-secret-manager gcs gcs-slim github-documents glue grafana hana hex hive hive-metastore iceberg iceberg-catalog informatica informix json-schema kafka kafka-connect kinesis ldap looker lookml mariadb matillion-dpc metabase microstrategy mlflow mode mongodb mssql mssql-odbc mysql neo4j nifi notion odcs okta omni oracle pinecone postgres powerbi powerbi-report-server preset presto presto-on-hive pulsar qlik-sense quicksight rdf redash redshift redshift-slim s3 s3-slim sac sagemaker salesforce sap-datasphere sigma slack snaplogic snowflake snowflake-queries snowflake-slim snowflake-summary snowplow sql-parser sql-queries sqlalchemy starburst-trino-usage starrocks superset sync-file-emitter tableau teradata thoughtspot tidb timescaledb trino unity-catalog unstructured vertexai vertica all cloud dev docs lint testing-utils integration-tests debug

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acryl_datahub-1.7.0.9rc1.tar.gz (4.5 MB view details)

Uploaded Sep 3, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

acryl_datahub-1.7.0.9rc1-py3-none-any.whl (5.3 MB view details)

Uploaded Sep 3, 2026 Python 3

File details

Details for the file acryl_datahub-1.7.0.9rc1.tar.gz.

File metadata

Download URL: acryl_datahub-1.7.0.9rc1.tar.gz
Upload date: Sep 3, 2026
Size: 4.5 MB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.10.21

File hashes

Hashes for acryl_datahub-1.7.0.9rc1.tar.gz
Algorithm	Hash digest
SHA256	`7851ad5b01608a43c87461e9f13432f468d0d9f25cd7e3c9a20fb7948cf6ce45`
MD5	`3aa55493d9280ab733c3299296b7f6a0`
BLAKE2b-256	`5011344fcdc9f46bac19e9db8c89308beabbdffb2cbfb5f3c7172f3c17ada23e`

See more details on using hashes here.

File details

Details for the file acryl_datahub-1.7.0.9rc1-py3-none-any.whl.

File metadata

Download URL: acryl_datahub-1.7.0.9rc1-py3-none-any.whl
Upload date: Sep 3, 2026
Size: 5.3 MB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.10.21

File hashes

Hashes for acryl_datahub-1.7.0.9rc1-py3-none-any.whl
Algorithm	Hash digest
SHA256	`cd14723ad9aff6e828885b9665d24b6d91557bf642b3bf6f07cf1fcf1a167280`
MD5	`9511b140d69085fd53781d76ee325f71`
BLAKE2b-256	`1732d3cc51838d0f9960ee85fe88c7cc07abaefb5f79567ea16ced8d1dd5dbda`