Apache Airflow Kettle Operator
KettleOperator which consists of KettleRunJobOperator and KettleRunTransformationOperator responsible for running Hitachi Vantara's PDI (Pentaho Data Integration) jobs and transformations from .kjb and .ktr files.
Currently, there's no support for CarteServer and the only way to make it work is to make sure that PDI is deployed within the same container as your Airflow installation and the operator is deployed locally (no repository support either, yet).
Requirements
- Python 3.7+
- Apache Airflow 2.0+
- Hitachi Vantara PDI (within the same container)
Setup
Installation
Installation can be done via pip, and the package itself will be installed in your site-packages directory (just like any other pip installation).
python -m site to find out where are your site-packages installed. This package will be located in the kettle_provider directory.
pip install apache-airflow-providers-kettle
Usage
To use the Operators you first have to import them within your Airflow .py files (presumably your DAG).
from kettle_provider.operators.kettle_operator import KettleRunJobOperator, KettleRunTransformationOperator
The Operators can then be used just like any other.
run_job = KettleRunJobOperator(
task_id='kettle-run-job',
file='test-job.kjb',
params={
'test-parameter-1': 'test-value-1',
'test-parameter'2': 'test-value-2',
},
)
run_transformation = KettleRunTransformationOperator(
task_id='kettle-run-transformation',
file='test-job.ktr',
params={
'test-parameter-1': 'test-value-1',
'test-parameter'2': 'test-value-2',
},
)
Available parameters
Below are the parameters you can use when defining the tasks with their default values.
Below list excludes base parameters inherited from BaseOperator class (such as task_id, etc.)
KettleRunJobOperator(
pdipath: str = '/opt/pentaho/data-integration/' # PDI installation
filepath: str = '/opt/pentaho/data-integration/jobs/' # PDI jobs directory
file: str | None = None # .kjb file to run
logfile: str = '/opt/pentaho/data-integration/logs/pdi.kitchen.log', # logfile for kitchen runs
maxloglines: int = 0, # max log lines for kitchen logfile (0 = no limit)
maxlogtimeout: int = 0, # max log age in seconds for kitchen logfile (0 = no limit)
loglevel: str = 'Basic', # log level (Basic, Detailed, Debug, Rowlevel, Error, Nothing)
params: dict[str, str] | None = None, # dictionary of parameters
output_encoding: str = 'utf-8', # output encoding for exit commands
**kwargs
)
KettleRunTransformationOperator(
pdipath: str = '/opt/pentaho/data-integration/' # PDI installation
filepath: str = '/opt/pentaho/data-integration/transformations/' # PDI jobs directory
file: str | None = None # .ktr file to run
logfile: str = '/opt/pentaho/data-integration/logs/pdi.pan.log', # logfile for pan runs
maxloglines: int = 0, # max log lines for kitchen logfile (0 = no limit)
maxlogtimeout: int = 0, # max log age in seconds for kitchen logfile (0 = no limit)
loglevel: str = 'Basic', # log level (Basic, Detailed, Debug, Rowlevel, Error, Nothing)
params: dict[str, str] | None = None, # dictionary of parameters
output_encoding: str = 'utf-8', # output encoding for exit commands
**kwargs
)
To-dos
- CarteServer support (for PDI deployed in a different container)
- PDI Repository support
- Better exit codes (currently all we get are bash exit-codes)
- Support for Windows machines (currently, the commands are all executed with bash)
Metadata
Release files for apache-airflow-providers-kettle 1.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| apache_airflow_providers_kettle-1.0.7.tar.gz | 8.8 kB | Without provenance | Details for apache_airflow_providers_kettle-1.0.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| apache_airflow_providers_kettle-1.0.7-py3-none-any.whl | Python 3 | none | any | Details for apache_airflow_providers_kettle-1.0.7-py3-none-any.whl |
Total release size: 18.6 kB
Release files / apache_airflow_providers_kettle-1.0.7.tar.gz
| Download URL | apache_airflow_providers_kettle-1.0.7.tar.gz |
|---|---|
| Size | 8.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f6f950345da8c3f8ce941d15fa6985348fa8f04898fc62986638203a8a590784
|
|
BLAKE2b-256 checksum How to use checksums |
c81869e27106db0b589e2f99cf34c563d91d295a0d6663887e2353dbaa3e072e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.6
|
Release files / apache_airflow_providers_kettle-1.0.7-py3-none-any.whl
| Download URL | apache_airflow_providers_kettle-1.0.7-py3-none-any.whl |
|---|---|
| Size | 9.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
46027629d5760d36db8b1fcf3ac6f330685a1d38bca29600447dfa6eafd8f572
|
|
BLAKE2b-256 checksum How to use checksums |
8870720db3b09dfd1da6b5ecf90dab465afcb449dab8fc999c60f598a4f1034c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.6
|
Security for apache-airflow-providers-kettle 1.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.