Skip to main content

Apache Airflow Kettle Operator

PyPI PyPI - Downloads

KettleOperator which consists of KettleRunJobOperator and KettleRunTransformationOperator responsible for running Hitachi Vantara's PDI (Pentaho Data Integration) jobs and transformations from .kjb and .ktr files.

Currently, there's no support for CarteServer and the only way to make it work is to make sure that PDI is deployed within the same container as your Airflow installation and the operator is deployed locally (no repository support either, yet).

Requirements

  • Python 3.7+
  • Apache Airflow 2.0+
  • Hitachi Vantara PDI (within the same container)

Setup

Installation

Installation can be done via pip, and the package itself will be installed in your site-packages directory (just like any other pip installation).

python -m site to find out where are your site-packages installed. This package will be located in the kettle_provider directory.

pip install apache-airflow-providers-kettle

Usage

To use the Operators you first have to import them within your Airflow .py files (presumably your DAG).

from kettle_provider.operators.kettle_operator import KettleRunJobOperator, KettleRunTransformationOperator

The Operators can then be used just like any other.

run_job = KettleRunJobOperator(
    task_id='kettle-run-job',
    file='test-job.kjb',
    params={
        'test-parameter-1': 'test-value-1',
        'test-parameter'2': 'test-value-2',
    },
)
run_transformation = KettleRunTransformationOperator(
    task_id='kettle-run-transformation',
    file='test-job.ktr',
    params={
        'test-parameter-1': 'test-value-1',
        'test-parameter'2': 'test-value-2',
    },
)

Available parameters

Below are the parameters you can use when defining the tasks with their default values. Below list excludes base parameters inherited from BaseOperator class (such as task_id, etc.)

KettleRunJobOperator(
    pdipath: str = '/opt/pentaho/data-integration/'  # PDI installation
    filepath: str = '/opt/pentaho/data-integration/jobs/'  # PDI jobs directory
    file: str | None = None  # .kjb file to run
    logfile: str = '/opt/pentaho/data-integration/logs/pdi.kitchen.log',  # logfile for kitchen runs
    maxloglines: int = 0,  # max log lines for kitchen logfile (0 = no limit)
    maxlogtimeout: int = 0,  # max log age in seconds for kitchen logfile (0 = no limit)
    loglevel: str = 'Basic',  # log level (Basic, Detailed, Debug, Rowlevel, Error, Nothing)
    params: dict[str, str] | None = None,  # dictionary of parameters
    output_encoding: str = 'utf-8',  # output encoding for exit commands
    **kwargs
)
KettleRunTransformationOperator(
    pdipath: str = '/opt/pentaho/data-integration/'  # PDI installation
    filepath: str = '/opt/pentaho/data-integration/transformations/'  # PDI jobs directory
    file: str | None = None  # .ktr file to run
    logfile: str = '/opt/pentaho/data-integration/logs/pdi.pan.log',  # logfile for pan runs
    maxloglines: int = 0,  # max log lines for kitchen logfile (0 = no limit)
    maxlogtimeout: int = 0,  # max log age in seconds for kitchen logfile (0 = no limit)
    loglevel: str = 'Basic',  # log level (Basic, Detailed, Debug, Rowlevel, Error, Nothing)
    params: dict[str, str] | None = None,  # dictionary of parameters
    output_encoding: str = 'utf-8',  # output encoding for exit commands
    **kwargs
)

To-dos

  • CarteServer support (for PDI deployed in a different container)
  • PDI Repository support
  • Better exit codes (currently all we get are bash exit-codes)
  • Support for Windows machines (currently, the commands are all executed with bash)

Metadata

Release files for apache-airflow-providers-kettle 1.0.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for apache-airflow-providers-kettle 1.0.7
File Size Uploaded Provenance Actions
apache_airflow_providers_kettle-1.0.7.tar.gz 8.8 kB Without provenance Details for apache_airflow_providers_kettle-1.0.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for apache-airflow-providers-kettle 1.0.7
File Interpreter ABI Platform Actions
apache_airflow_providers_kettle-1.0.7-py3-none-any.whl Python 3 none any Details for apache_airflow_providers_kettle-1.0.7-py3-none-any.whl

Total release size: 18.6 kB

Release files / apache_airflow_providers_kettle-1.0.7.tar.gz

Download URL apache_airflow_providers_kettle-1.0.7.tar.gz
Size 8.8 kB
Tags Source
SHA-256 checksum
How to use checksums
f6f950345da8c3f8ce941d15fa6985348fa8f04898fc62986638203a8a590784
BLAKE2b-256 checksum
How to use checksums
c81869e27106db0b589e2f99cf34c563d91d295a0d6663887e2353dbaa3e072e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.6

Release files / apache_airflow_providers_kettle-1.0.7-py3-none-any.whl

Download URL apache_airflow_providers_kettle-1.0.7-py3-none-any.whl
Size 9.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
46027629d5760d36db8b1fcf3ac6f330685a1d38bca29600447dfa6eafd8f572
BLAKE2b-256 checksum
How to use checksums
8870720db3b09dfd1da6b5ecf90dab465afcb449dab8fc999c60f598a4f1034c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.6

Security for apache-airflow-providers-kettle 1.0.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page