acryl-executor 0.3.17

Run DataHub metadata ingestion tasks remotely via subprocess isolation with S3 log storage

Acryl Executor

Remote execution agent for running DataHub metadata ingestion tasks via subprocess isolation — with S3 log storage, plugin-based extensibility, and first-class support for the DataHub UI-triggered ingestion workflow.

Features

• Subprocess isolation — each ingestion run executes in a dedicated subprocess and virtual environment, preventing dependency conflicts between connectors
• UI-triggered ingestion — integrates directly with the DataHub UI to execute RUN_INGEST tasks on demand
• Connection testing — validate source/destination connectivity before running full ingestion via TEST_CONNECTION tasks
• S3 log storage — automatically compress and upload execution logs and artifacts to S3; optionally clean up local files after a successful upload
• Plugin architecture — register custom task implementations via Python entry points
• AWS Secrets Manager & GCP Secret Manager — built-in secret store plugins for retrieving credentials from AWS SM or GCP SM

Installation

pip install acryl-executor

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install acryl-executor

Task Types

Task	Description
RUN_INGEST (SubProcessIngestionTask)	Runs metadata ingestion in a subprocess; supports per-run DataHub versions and connector plugins
TEST_CONNECTION	Validates connectivity to a data source before ingestion

Cloud Logging (S3)

Set these environment variables to enable S3 log uploads:

Variable	Description
DATAHUB_CLOUD_LOG_BUCKET	S3 bucket to write logs to
DATAHUB_CLOUD_LOG_PATH	S3 path prefix for logs
DATAHUB_CLOUD_LOG_CLEANUP	Set true to remove local files after a successful upload (default: false)

Logs are tar-gzipped and stored at:

s3://<BUCKET>/<PATH>/<pipeline_id>/year=<Y>/month=<M>/day=<D>/<run_id>/

When cleanup is enabled, a .s3 sentinel file replaces each uploaded file, recording the S3 URI, upload timestamp, and original file size.

Plugin Registration

Custom tasks register via the datahub.executor.task.plugins entry point:

entry_points = {
    "datahub.executor.task.plugins": [
        "my_task = my_package.tasks:MyTask"
    ]
}

Links

• DataHub documentation
• Source code
• Changelog

License

Apache License 2.0