Skip to main content

Drag'N'Drop Web Frontend for Building and Managing Airflow DAGs

Project description

Windmill

Build Status

Drag'n'drop web app to manage and create Airflow DAGs. The goal is to have a Web UI that can generate YML Dag Definitions, integrating with custom operators and potentially existing DAGs. YML DAGs can then be synced to a remote repo

  • Front end is built using React on Typescript
  • Back end is built using Flask on Python 3.6+

Getting Started

  1. Install with pip install airflow-windmill
    1. Airflow is expected to be installed on the system
    2. Otherwise it can be packaged with windmill using pip install airflow-windmill[airflow]
  2. Run windmill init to create a local Windmill project
  3. cd windmill-project
  4. Run windmill run from this folder to run the app locally

MVP Required Features

Front-End Features

  • Dynamic Operators
  • Menu Dropdowns
  • Load Operators from App
  • Format operator display into classes
  • Search functionality for operators
  • Basic operator level properties
  • Implement DAG level properties
  • New DAG Functionality
  • Parameter Tooltips
  • Render arbitrary viewport windows for New/Save/Load etc
  • Overwrite/Save prompt on New
  • DAG renaming and save functionality
  • Open dag from menu
  • Switch nav menu to icons
  • Make input/output nodes more clear
  • Make save/load more efficient by removing non-essential values
  • Check if file already exists on rename
  • Prompt save if there are nodes on open
  • Fix loss of state on refresh bug
  • Put File details in File Browser
  • Make Flask Backend URI configurable
  • Add a last saved time to NavBar

Back-End Features

  • Generate Operator Lists
  • CLI to start Web and Front End
  • Generate DAG Spec
  • CLI to create new windmill project
  • CLI to start windmill from a windmill project
  • Implement windmill-dev start
  • Save/Load Windmill Files functionality
  • Get default values
  • Pull parameters from parent classes
  • Move airflow dependency as extra
  • ? Dedupe multi import operators - nothing preventing this but underlying issue is fixed
  • Convert WML into Python DAG
  • Get WML owner and last-modified details during wml list
  • Allow custom operators
  • Add defaults to CLI --help commands
  • Strategy for Python Opjects (e.g. callables) - maybe import statement?
  • Backport existing Python DAGs to WMLs
  • Allow YML updates to propogate to WMLs

Other features

  • Validate on backend or front end or both?
  • Doco

Dev User Guide

To run as a dev:

  1. Clone from git
  2. Run poetry install
  3. Run windmill-dev start-backend
  4. Run windmill-dev start-frontend

Future Usage Patterns

  • Auto-sync for windmill project to git

Deployment

cd scripts
sh build.sh {{PYPI_USER}} {{PYPI_PASS}}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-windmill-0.0.1.tar.gz (1.5 MB view hashes)

Uploaded Source

Built Distribution

airflow_windmill-0.0.1-py3-none-any.whl (1.5 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page