Skip to main content

Check for compatibility between Airflow versions

Project description

Apache Airflow Upgrade Check

PyPI version License PyPI - Python Version PyPI - Downloads Twitter Follow Slack Status

This package aims to easy the upgrade journey from Apache Airflow 1.10 to 2.0.

While we have put a lot of effort in to making this upgrade as painless as possible, with many changes providing upgrade path (where the old code continues to work and prints out a deprecation warning) there were unfortunately some breaking changes where we couldn't provide a compatibility shim.

The recommended upgrade path to get to Airflow 2.0.0 is to first upgrade to the latest release in the 1.10 series (at the time of writing: 1.10.14) and to then run this script.

pip install apache-airflow-upgrade-check
airflow upgrade_check

This will then print out a number of action items that you should follow before upgrading to 2.0.0 or above.

The exit code of the command will be 0 (success) if no problems are reported, or 1 otherwise.

For example:

============================================= STATUS =============================================

Legacy UI is deprecated by default......................................................SUCCESS
Users must set a kubernetes.pod_template_file value.....................................FAIL
Changes in import paths of hooks, operators, sensors and others.........................FAIL
Remove airflow.AirflowMacroPlugin class.................................................SUCCESS
Jinja Template Variables cannot be undefined............................................SUCCESS
Fernet is enabled by default............................................................FAIL
Logging configuration has been moved to new section.....................................SUCCESS
Connection.conn_id is not unique........................................................SUCCESS
GCP service account key deprecation.....................................................SUCCESS
Users must delete deprecated configs for KubernetesExecutor.............................FAIL
Changes in import path of remote task handlers..........................................SUCCESS
Chain between DAG and operator not allowed..............................................SUCCESS
SendGrid email uses old airflow.contrib module..........................................SUCCESS
Connection.conn_type is not nullable....................................................SUCCESS
Found 16 problems.

======================================== RECOMMENDATIONS =========================================

Users must set a kubernetes.pod_template_file value
---------------------------------------------------
In Airflow 2.0, KubernetesExecutor Users need to set a pod_template_file as a base
value for all pods launched by the KubernetesExecutor


Problems:

  1.  Please create a pod_template_file by running `airflow generate_pod_template`.
This will generate a pod using your aiflow.cfg settings

...

Additionally you can use "upgrade config" to:

  • specify rules you would like to ignore
  • extend the check using custom rules

For example:

airflow upgrade_check --config=/files/upgrade.yaml

the configuration file should be a proper yaml file similar to this one:

ignored_rules:
  - LegacyUIDeprecated
  - ConnTypeIsNotNullableRule
  - PodTemplateFileRule

custom_rules:
  - path.to.upgrade_module.VeryCustomCheckClass
  - path.to.upgrade_module.VeryCustomCheckClass2

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apache-airflow-upgrade-check-1.1.0rc1.tar.gz (31.3 kB view hashes)

Uploaded Source

Built Distribution

apache_airflow_upgrade_check-1.1.0rc1-py2.py3-none-any.whl (50.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page