Airflow to DolphinScheduler migration tool
Project description
airphin
Airflow to DolphinScheduler migration tool.
Installation
For now, it just for test and without publish to pypi but will be adding in the future. You could still install locally by yourself.
python pip install --upgrade airphin
Quick Start
Get example airflow DAG file via terminal:
wget https://raw.githubusercontent.com/WhaleOps/airphin/main/examples/airflow/tutorial.py
Convert it to Apache DolphinScheduler Python API define:
airphin tutorial.py
Support Statement
For now, we support following conversion from Airflow's DAG files
DAG
Before Conversion | After Conversion |
---|---|
from airflow import DAG |
from pydolphinscheduler.core.process_definition import ProcessDefinition |
with DAG(...) as dag: pass |
with ProcessDefinition(...) as dag: pass |
Operators
Dummy Operator
Before Conversion | After Conversion |
---|---|
from airflow.operators.dummy_operator import DummyOperator |
from pydolphinscheduler.tasks.shell import Shell |
from airflow.operators.dummy import DummyOperator |
from pydolphinscheduler.tasks.shell import Shell |
dummy = DummyOperator(...) |
dummy = Shell(..., command="echo 'airflow dummy operator'") |
Shell Operator
Before Conversion | After Conversion |
---|---|
from airflow.operators.bash import BashOperator |
from pydolphinscheduler.tasks.shell import Shell |
bash = BashOperator(...) |
bash = Shell(...) |
Spark Sql Operator
Before Conversion | After Conversion |
---|---|
from airflow.operators.spark_sql_operator import SparkSqlOperator |
from pydolphinscheduler.tasks.sql import Sql |
spark = SparkSqlOperator(...) |
spark = Sql(...) |
Python Operator
Before Conversion | After Conversion |
---|---|
from airflow.operators.python_operator import PythonOperator |
from pydolphinscheduler.tasks.python import Python |
python = PythonOperator(...) |
python = Python(...) |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
airphin-0.0.2.tar.gz
(14.7 kB
view hashes)
Built Distribution
airphin-0.0.2-py3-none-any.whl
(18.4 kB
view hashes)