Skip to main content

Dagsesh: Airflow session manager

Project description

Dagsesh: Airflow DAG Session Manager

Overview

An Apache Airflow session context manager that overrides the default AIRFLOW_HOME path with a random, ephemeral alternate.

Why is this useful? As per the Airflow configuration docs:

The first time you run Airflow, it will create a file called airflow.cfg in your $AIRFLOW_HOME directory (~/airflow by default).

Dagsesh delays the creation of AIRFLOW_HOME using a lazy-loading facility whilst injecting it with a random alternate. Great if you want to create a pristine Airflow environment for repeatable testing.

Dagsesh can be used as a convenient pytest plugin to prime an Airflow environment for testing.

top

Prerequisites

top

Getting Started

Makester is used as the Integrated Developer Platform.

(macOS Users) Upgrading GNU Make

Follow these notes to get GNU make.

Creating the Local Environment

Get the code and change into the top level git project directory:

git clone git@github.com:loum/dagsesh.git && cd dagsesh

NOTE: Run all commands from the top-level directory of the git repository.

For first-time setup, get the Makester project:

git submodule update --init

Initialise the environment:

make init-dev

Local Environment Maintenance

Keep Makester project up-to-date with:

git submodule update --remote --merge

top

Help

There should be a make target to get most things done. Check the help for more information:

make help

top

Running the Test Harness

We use pytest. To run the tests:

make tests

top

Using dagsesh Plugin in your Project's Test Suite

Add the dagsesh package as a dependency to your project's development environment so that the plugin is installed and visible in your PYTHONPATH. dagsesh takes care of the distribution's entry point so that pytest automatically finds the plugin module. Nothing else needs to be done.

NOTE: See Making your plugin installable by others for more information.


top

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagsesh-0.1.0.tar.gz (5.9 kB view hashes)

Uploaded Source

Built Distribution

dagsesh-0.1.0-py3-none-any.whl (6.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page