Acryl Executor
Remote execution agent for running DataHub metadata ingestion tasks via subprocess isolation — with S3 log storage, plugin-based extensibility, and first-class support for the DataHub UI-triggered ingestion workflow.
Features
- Subprocess isolation — each ingestion run executes in a dedicated subprocess and virtual environment, preventing dependency conflicts between connectors
- UI-triggered ingestion — integrates directly with the DataHub UI to execute
RUN_INGESTtasks on demand - Connection testing — validate source/destination connectivity before running full ingestion via
TEST_CONNECTIONtasks - S3 log storage — automatically compress and upload execution logs and artifacts to S3; optionally clean up local files after a successful upload
- Plugin architecture — register custom task implementations via Python entry points
- AWS Secrets Manager & GCP Secret Manager — built-in secret store plugins for retrieving credentials from AWS SM or GCP SM
Installation
pip install acryl-executor
Quick Start
python3 -m venv venv
source venv/bin/activate
pip install acryl-executor
Task Types
| Task | Description |
|---|---|
RUN_INGEST (SubProcessIngestionTask) |
Runs metadata ingestion in a subprocess; supports per-run DataHub versions and connector plugins |
TEST_CONNECTION |
Validates connectivity to a data source before ingestion |
Cloud Logging (S3)
Set these environment variables to enable S3 log uploads:
| Variable | Description |
|---|---|
DATAHUB_CLOUD_LOG_BUCKET |
S3 bucket to write logs to |
DATAHUB_CLOUD_LOG_PATH |
S3 path prefix for logs |
DATAHUB_CLOUD_LOG_CLEANUP |
Set true to remove local files after a successful upload (default: false) |
Logs are tar-gzipped and stored at:
s3://<BUCKET>/<PATH>/<pipeline_id>/year=<Y>/month=<M>/day=<D>/<run_id>/
When cleanup is enabled, a .s3 sentinel file replaces each uploaded file, recording the S3 URI, upload timestamp, and original file size.
Plugin Registration
Custom tasks register via the datahub.executor.task.plugins entry point:
entry_points = {
"datahub.executor.task.plugins": [
"my_task = my_package.tasks:MyTask"
]
}
Links
License
Apache License 2.0
Metadata
Release files for acryl-executor 0.3.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| acryl_executor-0.3.20.tar.gz | 56.3 kB | Without provenance | Details for acryl_executor-0.3.20.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| acryl_executor-0.3.20-py3-none-any.whl | Python 3 | none | any | Details for acryl_executor-0.3.20-py3-none-any.whl |
Total release size: 134.3 kB
Release files / acryl_executor-0.3.20.tar.gz
| Download URL | acryl_executor-0.3.20.tar.gz |
|---|---|
| Size | 56.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
045c204c376e1dd54e002223c3a405c00ee7a9529b71929c3f12e7c4f0e53b17
|
|
BLAKE2b-256 checksum How to use checksums |
b998ac67c7a8899674034034e98cd27155849f8aa0fd059c5c75c703ad478dee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.21
|
Release files / acryl_executor-0.3.20-py3-none-any.whl
| Download URL | acryl_executor-0.3.20-py3-none-any.whl |
|---|---|
| Size | 78.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d05406ae57e57316504d5ac06b5b637a7c6e8b5172687a62126b26cc7c86b748
|
|
BLAKE2b-256 checksum How to use checksums |
3abcaf67c09a7458f21ac3279fec51ebf893d2f39c11add8b17b3604dcdbbbc0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.21
|
Security for acryl-executor 0.3.20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.