Apache Airflow provider for KDBAirflowOperator
Project description
Airflow KDB Provider
A lightweight KDB provider for Apache Airflow, featuring the KDBAirflowOperator
. This provider allows for seamless integration between Airflow and KDB+/q, making it easier to automate data pipelines that involve KDB+/q.
Installation
You can install the Airflow KDB Provider package from PyPI using the following command:
pip install airflow-kdb-provider
Usage
To use the KDBAirflowOperator in your Airflow DAG, you must first import it and create an instance of the operator. Here is an example:
from airflow_kdb_provider.operators.kdb_operator import KDBOperator
kdb_operator = KDBOperator(
task_id='run_kdb_script',
command='/path/to/kdb_script.q',
params={'param1': 'value1', 'param2': 'value2'},
conn_id='kdb_conn',
dag=dag)
In this example, we create an instance of the KDBOperator and specify the following parameters:
task_id: the task ID for this operator command: the path to the KDB+/q script that we want to execute params: a dictionary of parameters that will be passed to the KDB+/q script as command-line arguments conn_id: the connection ID for the KDB+/q server that we want to use (this should be defined in Airflow's Connections interface) dag: the DAG that this operator belongs to Once you have created an instance of the KDBOperator, you can add it to your DAG like any other Airflow operator:
some_other_operator >> kdb_operator >> some_other_operator2
In this example, we have added the kdb_operator to our DAG and specified that it should be executed after some_other_operator and before some_other_operator2.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow-kdb-provider-0.1.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 437daf8b061acb873b9a05ec5e6bfb521488adada15b41cebcfbaead3b09b07d |
|
MD5 | 493a04f4c42150c2b270250f12996850 |
|
BLAKE2b-256 | c7c5fdbf5dfb000e0e2f00f0b4ee55c6140b39f8dfd305792c6faeabd0427f98 |
Hashes for airflow_kdb_provider-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57ff71fa22e39595a901501a7094873737430d77de9e3545cacf7f52650b480f |
|
MD5 | 5c304e228040a7de93180ea46c023a56 |
|
BLAKE2b-256 | 33e077cf2f66b34c315284748d7b3e2f40478ab344f3faea19fea85a788720ac |