Acryl Executor

Remote execution agent for running DataHub metadata ingestion tasks via subprocess isolation — with S3 log storage, plugin-based extensibility, and first-class support for the DataHub UI-triggered ingestion workflow.

Features

  • Subprocess isolation — each ingestion run executes in a dedicated subprocess and virtual environment, preventing dependency conflicts between connectors
  • UI-triggered ingestion — integrates directly with the DataHub UI to execute RUN_INGEST tasks on demand
  • Connection testing — validate source/destination connectivity before running full ingestion via TEST_CONNECTION tasks
  • S3 log storage — automatically compress and upload execution logs and artifacts to S3; optionally clean up local files after a successful upload
  • Plugin architecture — register custom task implementations via Python entry points
  • AWS Secrets Manager & GCP Secret Manager — built-in secret store plugins for retrieving credentials from AWS SM or GCP SM

Installation

pip install acryl-executor

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install acryl-executor

Task Types

Task Description
RUN_INGEST (SubProcessIngestionTask) Runs metadata ingestion in a subprocess; supports per-run DataHub versions and connector plugins
TEST_CONNECTION Validates connectivity to a data source before ingestion

Cloud Logging (S3)

Set these environment variables to enable S3 log uploads:

Variable Description
DATAHUB_CLOUD_LOG_BUCKET S3 bucket to write logs to
DATAHUB_CLOUD_LOG_PATH S3 path prefix for logs
DATAHUB_CLOUD_LOG_CLEANUP Set true to remove local files after a successful upload (default: false)

Logs are tar-gzipped and stored at:

s3://<BUCKET>/<PATH>/<pipeline_id>/year=<Y>/month=<M>/day=<D>/<run_id>/

When cleanup is enabled, a .s3 sentinel file replaces each uploaded file, recording the S3 URI, upload timestamp, and original file size.

Plugin Registration

Custom tasks register via the datahub.executor.task.plugins entry point:

entry_points = {
    "datahub.executor.task.plugins": [
        "my_task = my_package.tasks:MyTask"
    ]
}

Links

License

Apache License 2.0

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for acryl-data from gravatar.com acryl-data Avatar for shirshanka from gravatar.com shirshanka

License

OSI Approved, Apache Software License (Apache License 2.0)

Requires

Python >=3.10

Provides Extra

dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acryl_executor-0.3.20.tar.gz (56.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

acryl_executor-0.3.20-py3-none-any.whl (78.0 kB view details)

Uploaded Python 3

File details

Details for the file acryl_executor-0.3.20.tar.gz.

File metadata

  • Download URL: acryl_executor-0.3.20.tar.gz
  • Upload date:
  • Size: 56.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.10.21

File hashes

Hashes for acryl_executor-0.3.20.tar.gz
Algorithm Hash digest
SHA256 045c204c376e1dd54e002223c3a405c00ee7a9529b71929c3f12e7c4f0e53b17
MD5 a8a1e82d4269ddcac2470702b54322a1
BLAKE2b-256 b998ac67c7a8899674034034e98cd27155849f8aa0fd059c5c75c703ad478dee

See more details on using hashes here.

File details

Details for the file acryl_executor-0.3.20-py3-none-any.whl.

File metadata

File hashes

Hashes for acryl_executor-0.3.20-py3-none-any.whl
Algorithm Hash digest
SHA256 d05406ae57e57316504d5ac06b5b637a7c6e8b5172687a62126b26cc7c86b748
MD5 250f9b391ffcc0b09b5b6983f1b0b430
BLAKE2b-256 3abcaf67c09a7458f21ac3279fec51ebf893d2f39c11add8b17b3604dcdbbbc0

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

0.3.20 This release

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5.1

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

Pre-release

0.3.2rc1

2 files

0.3.1

2 files

0.3.0

2 files

0.2.11

2 files

0.2.10

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

Pre-release

0.2.7rc1

2 files

0.2.6

2 files

Pre-release

0.2.6rc1

2 files

Pre-release

0.2.6rc0

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

0.0.3.18

2 files

0.0.3.15

2 files

0.0.3.14

2 files

0.0.3.13

2 files

0.0.3.12

2 files

0.0.3.11

2 files

0.0.3.10

2 files

0.0.3.9

2 files

0.0.3.8

2 files

0.0.3.7

2 files

0.0.3.6

2 files

0.0.3.5

2 files

0.0.3.4

2 files

Pre-release

0.0.3.4rc2

2 files

Pre-release

0.0.3.4rc1

2 files

0.0.3.3

2 files

0.0.3.2

2 files

0.0.3.1

2 files

0.0.3

2 files

Pre-release

0.0.3rc8

2 files

Pre-release

0.0.3rc7

2 files

Pre-release

0.0.3rc6

2 files

Pre-release

0.0.3rc5

2 files

Pre-release

0.0.3rc4

2 files

Pre-release

0.0.3rc3

2 files

Pre-release

0.0.3rc2

2 files

0.0.2.5

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page