Acryl Executor

Remote execution agent for running DataHub metadata ingestion tasks via subprocess isolation — with S3 log storage, plugin-based extensibility, and first-class support for the DataHub UI-triggered ingestion workflow.

Features

Subprocess isolation — each ingestion run executes in a dedicated subprocess and virtual environment, preventing dependency conflicts between connectors
UI-triggered ingestion — integrates directly with the DataHub UI to execute RUN_INGEST tasks on demand
Connection testing — validate source/destination connectivity before running full ingestion via TEST_CONNECTION tasks
S3 log storage — automatically compress and upload execution logs and artifacts to S3; optionally clean up local files after a successful upload
Plugin architecture — register custom task implementations via Python entry points
AWS Secrets Manager & GCP Secret Manager — built-in secret store plugins for retrieving credentials from AWS SM or GCP SM

Installation

pip install acryl-executor

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install acryl-executor

Task Types

Task	Description
`RUN_INGEST` (`SubProcessIngestionTask`)	Runs metadata ingestion in a subprocess; supports per-run DataHub versions and connector plugins
`TEST_CONNECTION`	Validates connectivity to a data source before ingestion

Cloud Logging (S3)

Set these environment variables to enable S3 log uploads:

Variable	Description
`DATAHUB_CLOUD_LOG_BUCKET`	S3 bucket to write logs to
`DATAHUB_CLOUD_LOG_PATH`	S3 path prefix for logs
`DATAHUB_CLOUD_LOG_CLEANUP`	Set `true` to remove local files after a successful upload (default: `false`)

Logs are tar-gzipped and stored at:

s3://<BUCKET>/<PATH>/<pipeline_id>/year=<Y>/month=<M>/day=<D>/<run_id>/

When cleanup is enabled, a .s3 sentinel file replaces each uploaded file, recording the S3 URI, upload timestamp, and original file size.

Plugin Registration

Custom tasks register via the datahub.executor.task.plugins entry point:

entry_points = {
    "datahub.executor.task.plugins": [
        "my_task = my_package.tasks:MyTask"
    ]
}

License

Apache License 2.0

Release files for acryl-executor 0.3.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acryl-executor 0.3.20
File	Size	Uploaded
acryl_executor-0.3.20.tar.gz	56.3 kB	Sep 3, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for acryl-executor 0.3.20
File	Interpreter	ABI	Platform	Reset
acryl_executor-0.3.20-py3-none-any.whl 78.0 kB Sep 3, 2026	Python 3	none	any	Details

Total release size: 134.3 kB

Release files / acryl_executor-0.3.20.tar.gz

Download URL	acryl_executor-0.3.20.tar.gz
Size	56.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`045c204c376e1dd54e002223c3a405c00ee7a9529b71929c3f12e7c4f0e53b17`
BLAKE2b-256 checksum How to use checksums	`b998ac67c7a8899674034034e98cd27155849f8aa0fd059c5c75c703ad478dee`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.21`

Release files / acryl_executor-0.3.20-py3-none-any.whl

Download URL	acryl_executor-0.3.20-py3-none-any.whl
Size	78.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`d05406ae57e57316504d5ac06b5b637a7c6e8b5172687a62126b26cc7c86b748`
BLAKE2b-256 checksum How to use checksums	`3abcaf67c09a7458f21ac3279fec51ebf893d2f39c11add8b17b3604dcdbbbc0`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.21`