This release is a pre-release and may not be stable for production use.
DataHub SDK and CLI
DataHub's ingestion framework and CLI — pull metadata from 50+ data sources into your catalog, or push it programmatically from your own pipelines and applications.
Pull-based integrations crawl your data systems on a schedule: Snowflake, BigQuery, dbt, Looker, Airflow, and many more.
Push-based integrations let you emit metadata directly from code as it happens: Python SDK, Java SDK, Spark, Great Expectations, and others.
What you can do
- Pull metadata from databases, warehouses, BI tools, and orchestrators using 50+ ready-made connectors
- Push metadata programmatically using the Python or Java SDK
- Transform and filter metadata in transit using built-in transformers
- Schedule and manage ingestion pipelines via CLI or the DataHub UI
- Automate lineage, ownership, tags, and documentation across your data assets
Supported sources
Snowflake · BigQuery · Redshift · dbt · Databricks · Looker · Tableau · Power BI · Airflow · Spark · Kafka · PostgreSQL · MySQL · Hive · Glue · S3 · Iceberg · Unity Catalog · Sigma · Mode · Superset · Metabase · and many more
Installation
pip install acryl-datahub
datahub version
Quickstart
Pull metadata from a source (recipe)
# snowflake_recipe.yml
source:
type: snowflake
config:
account_id: my_account
username: my_user
password: my_password
role: DATAHUB_ROLE
warehouse: COMPUTE_WH
sink:
type: datahub-rest
config:
server: http://localhost:8080
datahub ingest -c snowflake_recipe.yml
Emit metadata from code (SDK)
from datahub.sdk import DataHubClient, Dataset
client = DataHubClient.from_env()
dataset = Dataset(platform="snowflake", name="mydb.schema.table")
dataset.set_description("My table description")
dataset.set_owners(["urn:li:corpuser:jane"])
client.entities.upsert(dataset)
Ingest via the DataHub UI
No CLI required — configure and run ingestion directly from the DataHub UI under Ingestion → Create new source.
Ingestion methods
| Method | Best for |
|---|---|
| CLI + YAML recipe | Scheduled batch ingestion, CI/CD pipelines |
| Python SDK | Programmatic or event-driven metadata emission |
| Java SDK | JVM-based integrations (Spark, Flink, etc.) |
| UI Ingestion | One-click setup, no code required |
Key CLI commands
datahub init # Connect to a DataHub instance (interactive)
datahub init --username datahub --password datahub # Quickstart with local defaults
datahub ingest -c recipe.yml # Run an ingestion pipeline
datahub search "my table" # Search for entities by keyword
datahub search "*" --filter platform=snowflake --filter entity_type=dataset
datahub graphql --list-operations # Explore all available GraphQL operations
datahub graphql --query "{ me { corpUser { urn } } }" # Run a GraphQL query
datahub get --urn <urn> # Fetch metadata for an entity
datahub delete --urn <urn> # Delete a metadata entity
datahub timeline --urn <urn> # View metadata change history
Links
2 maintainers
Metadata
2 maintainers
Release files for acryl-datahub 1.7.0.18rc1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| acryl_datahub-1.7.0.18rc1.tar.gz | 5.4 MB | Without provenance | Details for acryl_datahub-1.7.0.18rc1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| acryl_datahub-1.7.0.18rc1-py3-none-any.whl | Python 3 | none | any | Details for acryl_datahub-1.7.0.18rc1-py3-none-any.whl |
Total release size: 11.6 MB
Release files / acryl_datahub-1.7.0.18rc1.tar.gz
| Download URL | acryl_datahub-1.7.0.18rc1.tar.gz |
|---|---|
| Size | 5.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad9aeac39e6011f9389e0df574e1cb395d5b0da1dc8abce1079a69632b247d9d
|
|
BLAKE2b-256 checksum How to use checksums |
28e260b1540b095e5d1f6ad1a1062c2a23c3bcc9df853eddcf601e73b9a3655c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / acryl_datahub-1.7.0.18rc1-py3-none-any.whl
| Download URL | acryl_datahub-1.7.0.18rc1-py3-none-any.whl |
|---|---|
| Size | 6.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9d8cf5224188f6c425d8e5c3026509d5155e1731bfab8cda23a903a189687853
|
|
BLAKE2b-256 checksum How to use checksums |
25a35ba77ade0fbc4396ebb971d8ca7e46864f81d1084ed1b8bcb2b90cce53fb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for acryl-datahub 1.7.0.18rc1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
