Migration operators to assist moving variables, connections to Astronomer Cloud
Project description
Astronomer Starship Provider
Apache Airflow Provider containing Operators from Astronomer. The purpose of these operators is to better assist customers migrating to Astronomer hosted Airflow environments from MWAA, GCC, OSS. This provider is meant for MWAA 2.0.2 and Composer 1 since the plugin methods are unavailable.
Installation
Install and update using pip:
pip install astronomer-starship-provider
Starship
Starship is an Astronomer utility that assist end users with migrating Variables, Connections, and Environment Variables from their source Airflow environments to an Astronomer environment.
Usage
Add the following line to your requirements.txt in your source environment:
astronomer-starship-provider
Add the following DAG to your source environment:
from airflow import DAG
from astronomer.starship.operators import AstroMigrationOperator
from datetime import datetime
with DAG(
dag_id="astronomer_migration_dag",
start_date=datetime(2020, 8, 15),
schedule_interval=None,
) as dag:
AstroMigrationOperator(
task_id='export_meta',
deployment_url='{{ dag_run.conf["deployment_url"] }}',
token='{{ dag_run.conf["astro_token"] }}',
)
Deploy this DAG to your source Airflow environment, configured as described in the section below
Hit the Trigger DAG button in the Airflow UI when the DAG appears, and input the following in the configuration dictionary: - astro_token: To get user token for astronomer navigate to cloud.astronomer.io/token and login using your Astronomer credentials - deployment_url: To retrieve a deployment URL - navigate to the deployment that you’d like to migrate to in the Astronomer UI, click Open Airflow and copy the page URL (excluding /home on the end of the URL)
Once the DAG successfully runs, your connections, variables, and environment variables should all be migrated to Astronomer
Configuration
The AstroMigrationOperator can be configured as follows: - You can update the list of environment variable names under the env_include_list parameter that need to be migrated to Astronomer. None are migrated by default. - if there are any Airflow Variables or Airflow Connections that should NOT be migrated, add them to the variable_exclude_list & connection_exclude_list parameters.
AstroMigrationOperator(
task_id='export_meta',
deployment_url='{{ dag_run.conf["deployment_url"] }}',
token='{{ dag_run.conf["astro_token"] }}',
variables_exclude_list=["deployment_url", "astro_token"],
connection_exclude_list=["some_conn_1"],
env_include_list=["FOO", "BAR"]
)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for astronomer-starship-provider-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3cbc8f4bf83ea9a6a42fc6b6ce31572bb0ccc7e605358c5e29ab4e222230d63c |
|
MD5 | 72b383f2fe01c73f8580741ce27f8992 |
|
BLAKE2b-256 | d8b7f11fb588bca449e233ae3cec35dcb674f8596ba22a2dc6b0d6eb3c57c0d3 |