python-to-yaml-dag-converter-mwaa-serverless 0.1.0

CLI tool that converts Python airflow Dags to YAML Dags for use with dagfactory

Python to Yaml Dag Converter for MWAA Serverless

Python to Yaml Dag Converter for MWAA Serverless is a command-line tool for converting Apache Airflow® Dags written in Python to dag-factory compatible Dags written in YAML.

Prerequisites

• Python 3.11+

Benefits

• Onboard existing Python Dags to declarative YAML
• Ensures compatibility with DagFactory 1.0
• Convert Python Dags to MWAA Serverless compatible YAML
• Upload converted YAML to S3

⚡ Quick Start

Step 1: Install

pip install python-to-yaml-dag-converter-mwaa-serverless

Step 2: Use

dag-converter convert <python-dag-file>

Call with the --help flag for more options

Step 3: Review

Review the converted Dags in the output_yaml folder in the current directory

Options

• --help Show help message and exit
• --output: PATH Path to output converted YAML Dag(s) to [default: output_yaml]
• --bucket: TEXT S3 bucket to upload converted Dags to. Uses local AWS credentials
• --validate/--no-validate: Validate the output YAML using DagFactory
• --debug/--no-debug Enable logging DagBag objects to .initial_task_attributes before and after conversion

Limitations

• Only AWS operators and AWS provider packages are supported
• Behavior that is not supported in dag-factory will not be converted
• Dynamic task mapping conversion is not supported

Examples

• See supported examples
• Try one out

dag-converter convert airflow_s3.py

Contributing and Bug Reporting

See CONTRIBUTING for more information.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.