datahub-custom-transformers 0.2.1

Collection of custom DataHub transformers for metadata enhancement

DataHub Custom Transformers

A collection of custom DataHub transformers for various metadata enhancement tasks.

Features

• 🏗️ Modular Design: Easy to add new transformers
• 🔧 Production Ready: Tested and documented transformers
• 🔌 Auto-Discovery: Transformers are automatically registered with DataHub

Installation

uv add datahub-custom-transformers

Available Transformers

Domain Structured Properties Transformer

Adds domain-type structured properties to all datasets in an ingestion.

Use Case: Organizational data classification where all datasets from a source belong to the same environment, team, or department.

transformers:
  - type: "simple_add_dataset_domain_structured_properties"
    config:
      properties:
        environment: "production_environment"
        team: "data_engineering_team"
        department: "engineering_department"

Quick Start

1. Prerequisites

Create structured properties in DataHub:

# structured_properties.yaml
- id: department
  type: urn
  description: "Data environment assignment"
  display_name: "Environment"
  entity_types: [dataset]
  cardinality: SINGLE
  type_qualifier:
    allowed_types: ["urn:li:entityType:datahub.domain"]

Create domain entities:

• production_environment
• data_engineering_team

2. Use in Ingestion Recipe

source:
  type: postgres
  config:
    host_port: "localhost:5432"
    database: "analytics_db"

transformers:
  - type: "simple_add_dataset_domain_structured_properties"
    config:
      properties:
        environment: "production_environment"
        team: "data_engineering_team"

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

3. Run Ingestion

datahub ingest -c config.yaml

Result

All datasets will have structured properties:

{
  "structuredProperties": {
    "properties": [
      {
        "propertyUrn": "urn:li:structuredProperty:environment",
        "values": ["urn:li:domain:production_environment"]
      },
      {
        "propertyUrn": "urn:li:structuredProperty:team",
        "values": ["urn:li:domain:data_engineering_team"]
      }
    ]
  }
}

Supported DataHub Sources

Works with all DataHub sources:

• BigQuery, Snowflake, PostgreSQL, MySQL, Redshift
• dbt, Airflow, Kafka, S3
• And many more...

Requirements

• Python 3.11+
• acryl-datahub >= 0.12.0

Contributing

1. Fork the repository
2. Create a feature branch
3. Add your transformer with tests
4. Submit a pull request

Support

• 📖 Documentation
• 🐛 Issues
• 💬 Discussions

License

MIT License - see LICENSE file.