Automate migration from Airflow 1 to Airflow 2
Project description
Astronomer Nova
(Formerly Airflow V2 Upgrader)
This project aims to automate common tasks when upgrading DAGs from Airflow 1 to Airflow 2, making use of RedBaron to parse and manipulate the Python syntax tree.
Features
- Automatically upgrade DAG source code from Airflow 1 to Airflow 2
- Scan Dags and report changes (Coming Soon)
Installation
You can install the Astronomer Nova CLI through Pip
pip install astronomer-nova
The simplest way to get started with Astronomer Nova is by using the astro cli.
Simply add astronomer-nova
to your list of dependencies in requirements.txt
.
Then you can run it with the following commands:
astro dev start
astro dev bash
nova
Installation from sources
To install astronomer-nova in development mode:
pip install --editable .[lint,test]
To build from sources:
python -m build
OR
python -m pip wheel .
Running tests:
python -m pytest
How to Use
nova dags/ dags/upgraded/
For more information, you can access the help menu with:
nova --help
How it works
Astronomer Nova performs the upgrade by:
- Building an import map for classes in
airflow.hooks
,airflow.operators
,airflow.sensors
, andairflow.providers
, which maps the classes to the current Airflow version import path. - Modifying Python files in a DAG directory to update
- Imports
- DAG access controls
- XCOM push
- XCOM pull
- Writing updated DAG to either the same DAG directory or a parameter output directory with
_upgraded
added to the filename.
The script is configured to run against ./dags
and to write output to ./dags/upgraded
by default.
Caveats
In order to load the Airflow import map, Airflow provider packages must be installed. There are often cross-dependencies within Airflow providers (eg airflow.operators.s3_to_hive_operator
required the Hive provider), so make sure the requirements.txt
file includes all necessary .
The script currently tries to correct imports that do not need to be changed going from Airflow 1 to 2, and logging does not include which files are currently being acted against.
The code uses Astronomer-centric opinions, like removing DAG-level RBAC. In the future, I'd rather have these decisions configurable at runtime, as well as generating per-DAG reports on the DAGs that can be used for further remediation, as well as understanding of DAG requirements, like the use of Connections and Variables.
Contribute
- Issue Tracker: https://github.com/astronomer/astronomer-nova/issues
- Source Code: https://github.com/astronomer/astronomer-nova
Contact
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for astronomer_nova-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cb8f2cdd7db4df3962f25c868c61a649ea3cdeaaf7351ef193a81a3f804a135 |
|
MD5 | 201aa0967d19922bb9da36e4ea4f9a22 |
|
BLAKE2b-256 | bf3a8f85b63abf63c917f10780bf03a02ae738d7fa1171dabadfbfd143d28f8e |