Astronomer Nova

(Formerly Airflow V2 Upgrader)

This project aims to automate common tasks when upgrading DAGs from Airflow 1 to Airflow 2, making use of RedBaron to parse and manipulate the Python syntax tree.

Features

• Automatically upgrade DAG source code from Airflow 1 to Airflow 2
• Scan Dags and report changes (Coming Soon)

Installation

You can install the Astronomer Nova CLI through Pip

pip install astronomer-nova

The simplest way to get started with Astronomer Nova is by using the astro cli. Simply add astronomer-nova to your list of dependencies in requirements.txt. Then you can run it with the following commands:

astro dev start
astro dev bash
nova

Installation from sources

To install astronomer-nova in development mode:

pip install --editable .[lint,test]

To build from sources:

python -m build

OR

python -m pip wheel .

Running tests:

python -m pytest

How to Use

nova dags/ dags/upgraded/

For more information, you can access the help menu with:

nova --help

How it works

Astronomer Nova performs the upgrade by:

1. Building an import map for classes in airflow.hooks, airflow.operators, airflow.sensors, and airflow.providers, which maps the classes to the current Airflow version import path.
2. Modifying Python files in a DAG directory to update
   • Imports
   • DAG access controls
   • XCOM push
   • XCOM pull
3. Writing updated DAG to either the same DAG directory or a parameter output directory with _upgraded added to the filename.

The script is configured to run against ./dags and to write output to ./dags/upgraded by default.

Caveats

In order to load the Airflow import map, Airflow provider packages must be installed. There are often cross-dependencies within Airflow providers (eg airflow.operators.s3_to_hive_operator required the Hive provider), so make sure the requirements.txt file includes all necessary .

The script currently tries to correct imports that do not need to be changed going from Airflow 1 to 2, and logging does not include which files are currently being acted against.

The code uses Astronomer-centric opinions, like removing DAG-level RBAC. In the future, I'd rather have these decisions configurable at runtime, as well as generating per-DAG reports on the DAGs that can be used for further remediation, as well as understanding of DAG requirements, like the use of Connections and Variables.

Contribute

• Issue Tracker: https://github.com/astronomer/astronomer-nova/issues
• Source Code: https://github.com/astronomer/astronomer-nova

Contact

cabella@astronomer.io