Acryl Executor
Remote execution agent for running DataHub metadata ingestion tasks via subprocess isolation — with S3 log storage, plugin-based extensibility, and first-class support for the DataHub UI-triggered ingestion workflow.
Features
- Subprocess isolation — each ingestion run executes in a dedicated subprocess and virtual environment, preventing dependency conflicts between connectors
- UI-triggered ingestion — integrates directly with the DataHub UI to execute
RUN_INGESTtasks on demand - Connection testing — validate source/destination connectivity before running full ingestion via
TEST_CONNECTIONtasks - S3 log storage — automatically compress and upload execution logs and artifacts to S3; optionally clean up local files after a successful upload
- Plugin architecture — register custom task implementations via Python entry points
- AWS Secrets Manager & GCP Secret Manager — built-in secret store plugins for retrieving credentials from AWS SM or GCP SM
Installation
pip install acryl-executor
Quick Start
python3 -m venv venv
source venv/bin/activate
pip install acryl-executor
Task Types
| Task | Description |
|---|---|
RUN_INGEST (SubProcessIngestionTask) |
Runs metadata ingestion in a subprocess; supports per-run DataHub versions and connector plugins |
TEST_CONNECTION |
Validates connectivity to a data source before ingestion |
Cloud Logging (S3)
Set these environment variables to enable S3 log uploads:
| Variable | Description |
|---|---|
DATAHUB_CLOUD_LOG_BUCKET |
S3 bucket to write logs to |
DATAHUB_CLOUD_LOG_PATH |
S3 path prefix for logs |
DATAHUB_CLOUD_LOG_CLEANUP |
Set true to remove local files after a successful upload (default: false) |
Logs are tar-gzipped and stored at:
s3://<BUCKET>/<PATH>/<pipeline_id>/year=<Y>/month=<M>/day=<D>/<run_id>/
When cleanup is enabled, a .s3 sentinel file replaces each uploaded file, recording the S3 URI, upload timestamp, and original file size.
Plugin Registration
Custom tasks register via the datahub.executor.task.plugins entry point:
entry_points = {
"datahub.executor.task.plugins": [
"my_task = my_package.tasks:MyTask"
]
}
Links
License
Apache License 2.0
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file acryl_executor-0.3.18.tar.gz.
File metadata
- Download URL: acryl_executor-0.3.18.tar.gz
- Upload date:
- Size: 55.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2a7ea40ab93e245321083a02278922a2b927673d83868a42f9ddd93cd4caecf3
|
|
| MD5 |
fe715c77d797a6b0101c0222ed1d1b1f
|
|
| BLAKE2b-256 |
5e403f41f009de08791f160314b95cf4a05128e77afb6d0f601ef60aaa3fcdec
|
File details
Details for the file acryl_executor-0.3.18-py3-none-any.whl.
File metadata
- Download URL: acryl_executor-0.3.18-py3-none-any.whl
- Upload date:
- Size: 77.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2539c47bf9ec64ab2101c4c863d398bb1f1506cbd5cbd39aa2c449cbf67098f1
|
|
| MD5 |
19da2f0cd697a024fcd88f6c23531400
|
|
| BLAKE2b-256 |
a730e63e32127316578e835b7280a6777ec57492fbaf93cee7391e228b2d2d1d
|
