Dagsesh: Airflow DAG Session Manager

• Overview
• Prerequisites
• Getting Started
  • (macOS Users) Upgrading GNU Make
  • Creating the Local Environment
  • Local Environment Maintenance
• Help
• Running the Test Harness
• Using dagsesh Plugin in your Project's Test Suite
• FAQs

Overview

An Apache Airflow session context manager that overrides the default AIRFLOW_HOME path with a random, ephemeral alternate.

Why is this useful? As per the Airflow configuration docs:

The first time you run Airflow, it will create a file called airflow.cfg in your $AIRFLOW_HOME directory (~/airflow by default).

Dagsesh delays the creation of AIRFLOW_HOME using a lazy-loading facility whilst injecting it with a random alternate. Great if you want to create a pristine Airflow environment for repeatable testing.

Dagsesh can be used as a convenient pytest plugin to prime an Airflow environment for testing.

top

Prerequisites

• GNU make
• Python 3 Interpreter. We recommend installing pyenv
• Docker

top

Getting Started

Makester is used as the Integrated Developer Platform.

(macOS Users) Upgrading GNU Make

Follow these notes to get GNU make.

Creating the Local Environment

Get the code and change into the top level git project directory:

git clone git@github.com:loum/dagsesh.git && cd dagsesh

NOTE: Run all commands from the top-level directory of the git repository.

For first-time setup, get the Makester project:

git submodule update --init

Initialise the environment:

make init-dev

Local Environment Maintenance

Keep Makester project up-to-date with:

git submodule update --remote --merge

top

Help

There should be a make target to get most things done. Check the help for more information:

make help

top

Running the Test Harness

We use pytest. To run the tests:

make tests

top

Using dagsesh Plugin in your Project's Test Suite

Add the dagsesh package as a dependency to your project's development environment so that the plugin is installed and visible in your PYTHONPATH. dagsesh takes care of the distribution's entry point so that pytest automatically finds the plugin module. Nothing else needs to be done.

NOTE: See Making your plugin installable by others for more information.

top