This release is a pre-release and may not be stable for production use.
DataHub SDK and CLI
DataHub's ingestion framework and CLI — pull metadata from 50+ data sources into your catalog, or push it programmatically from your own pipelines and applications.
Pull-based integrations crawl your data systems on a schedule: Snowflake, BigQuery, dbt, Looker, Airflow, and many more.
Push-based integrations let you emit metadata directly from code as it happens: Python SDK, Java SDK, Spark, Great Expectations, and others.
What you can do
- Pull metadata from databases, warehouses, BI tools, and orchestrators using 50+ ready-made connectors
- Push metadata programmatically using the Python or Java SDK
- Transform and filter metadata in transit using built-in transformers
- Schedule and manage ingestion pipelines via CLI or the DataHub UI
- Automate lineage, ownership, tags, and documentation across your data assets
Supported sources
Snowflake · BigQuery · Redshift · dbt · Databricks · Looker · Tableau · Power BI · Airflow · Spark · Kafka · PostgreSQL · MySQL · Hive · Glue · S3 · Iceberg · Unity Catalog · Sigma · Mode · Superset · Metabase · and many more
Installation
pip install acryl-datahub
datahub version
Quickstart
Pull metadata from a source (recipe)
# snowflake_recipe.yml
source:
type: snowflake
config:
account_id: my_account
username: my_user
password: my_password
role: DATAHUB_ROLE
warehouse: COMPUTE_WH
sink:
type: datahub-rest
config:
server: http://localhost:8080
datahub ingest -c snowflake_recipe.yml
Emit metadata from code (SDK)
from datahub.sdk import DataHubClient, Dataset
client = DataHubClient.from_env()
dataset = Dataset(platform="snowflake", name="mydb.schema.table")
dataset.set_description("My table description")
dataset.set_owners(["urn:li:corpuser:jane"])
client.entities.upsert(dataset)
Ingest via the DataHub UI
No CLI required — configure and run ingestion directly from the DataHub UI under Ingestion → Create new source.
Ingestion methods
| Method | Best for |
|---|---|
| CLI + YAML recipe | Scheduled batch ingestion, CI/CD pipelines |
| Python SDK | Programmatic or event-driven metadata emission |
| Java SDK | JVM-based integrations (Spark, Flink, etc.) |
| UI Ingestion | One-click setup, no code required |
Key CLI commands
datahub init # Connect to a DataHub instance (interactive)
datahub init --username datahub --password datahub # Quickstart with local defaults
datahub ingest -c recipe.yml # Run an ingestion pipeline
datahub search "my table" # Search for entities by keyword
datahub search "*" --filter platform=snowflake --filter entity_type=dataset
datahub graphql --list-operations # Explore all available GraphQL operations
datahub graphql --query "{ me { corpUser { urn } } }" # Run a GraphQL query
datahub get --urn <urn> # Fetch metadata for an entity
datahub delete --urn <urn> # Delete a metadata entity
datahub timeline --urn <urn> # View metadata change history
Links
2 maintainers
Metadata
2 maintainers
Release files for acryl-datahub 1.7.0.16rc1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| acryl_datahub-1.7.0.16rc1.tar.gz | 5.2 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| acryl_datahub-1.7.0.16rc1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.3 MB
Release files / acryl_datahub-1.7.0.16rc1.tar.gz
| Download URL | acryl_datahub-1.7.0.16rc1.tar.gz |
|---|---|
| Size | 5.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2e7bdc6d85313429120a795f0cb019e48e812db61900e3fe267f210a74f46c12
|
|
BLAKE2b-256 checksum How to use checksums |
588e26e27f1d1abb3d4ff67127e60921c9668b7f8e280ea41560649faad23151
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / acryl_datahub-1.7.0.16rc1-py3-none-any.whl
| Download URL | acryl_datahub-1.7.0.16rc1-py3-none-any.whl |
|---|---|
| Size | 6.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
484548226b471badb299e6a9432d6a8a267c44aa3def09ab4e8e22cc0c91eb32
|
|
BLAKE2b-256 checksum How to use checksums |
32129153ebdddac2de24c9e7e21a48540811cf639682b9c938c6fcaea9cc67c7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for acryl-datahub 1.7.0.16rc1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
