datahub-airbyte-source 0.2.0

Custom Airbyte source plugin for DataHub ingestion

datahub-airbyte-source

Custom Airbyte source plugin for DataHub ingestion.

What this provides

• Registers source.type: airbyte via DataHub plugin entry point.
• Extracts Airbyte connections as DataFlow + DataJob.
• Extracts recent sync jobs as DataProcessInstance events.
• Emits table-level lineage and optional column-level lineage when stream field mappings are present.

Install in DataHub ingestion

Use the package name in DataHub UI Advanced -> Extra Pip Libraries:

datahub-airbyte-source==0.2.0

Or install from a git URL/wheel URL.

Example recipe

source:
  type: airbyte
  config:
    api:
      # Auto detects Cloud when server_url host is api.airbyte.com.
      deployment_type: auto
      server_url: "https://api.airbyte.com"
      api_key: ${AIRBYTE_API_KEY}
    connection_patterns:
      allow:
        - ".*"
    history_sync_lookback_period: 7
    include_column_lineage: true

sink:
  type: datahub-rest
  config:
    server: "http://gms:8080"

Notes

• Supports both OSS /api/v1/* and Airbyte Cloud/public API endpoints.
• deployment_type can be auto, oss, or cloud.