Skip to main content

No project description provided

Project description

Windmill

Drag'n'drop web app to manage and create Airflow DAGs. The goal is to have a Web UI that can generate YML Dag Definitions, integrating with custom operators and potentially existing DAGs. YML DAGs can then be synced to a remote repo

  • Front end is built using Typescript React
  • Back end is built using Flask on Python 3.6+

MVP Required Features

Front-End Features

  • <input type="checkbox" checked="" disabled="" /> Dynamic Operators
  • <input type="checkbox" checked="" disabled="" /> Menu Dropdowns
  • <input type="checkbox" checked="" disabled="" /> Load Operators from App
  • <input type="checkbox" checked="" disabled="" /> Format operator display into classes
  • <input type="checkbox" checked="" disabled="" /> Search functionality for operators
  • <input type="checkbox" checked="" disabled="" /> Basic operator level properties
  • <input type="checkbox" checked="" disabled="" /> Implement DAG level properties
  • <input type="checkbox" checked="" disabled="" /> New DAG Functionality
  • <input type="checkbox" checked="" disabled="" /> Parameter Tooltips
  • <input type="checkbox" checked="" disabled="" /> Render arbitrary viewport windows for New/Save/Load etc
  • <input type="checkbox" checked="" disabled="" /> Overwrite/Save prompt on New
  • <input type="checkbox" checked="" disabled="" /> DAG renaming and save functionality
  • <input type="checkbox" disabled="" /> Open dag from menu
  • <input type="checkbox" disabled="" /> Check if file already exists on rename
  • <input type="checkbox" disabled="" /> Prompt save if there are nodes on open
  • <input type="checkbox" disabled="" /> Fix loss of state on refresh bug
  • <input type="checkbox" disabled="" /> Icons
  • <input type="checkbox" disabled="" /> Put File details in File Browser
  • <input type="checkbox" disabled="" /> Pull PORT from Flask host
  • <input type="checkbox" disabled="" /> Add a last saved time to NavBar

Back-End Features

  • <input type="checkbox" checked="" disabled="" /> Generate Operator Lists
  • <input type="checkbox" checked="" disabled="" /> CLI to start Web and Front End
  • <input type="checkbox" checked="" disabled="" /> Generate DAG Spec
  • <input type="checkbox" checked="" disabled="" /> CLI to create new windmill project
  • <input type="checkbox" checked="" disabled="" /> CLI to start windmill from a windmill project
  • <input type="checkbox" checked="" disabled="" /> Implement windmill-dev start
  • <input type="checkbox" checked="" disabled="" /> Save/Load Windmill Files functionality
  • <input type="checkbox" disabled="" /> Validate incoming WMLs (is there a need for this?)
  • <input type="checkbox" checked="" disabled="" /> Get default values
  • <input type="checkbox" checked="" disabled="" /> Pull parameters from parent classes
  • [?] Dedupe multi import operators - nothing preventing this but underlying issue is fixed
  • <input type="checkbox" disabled="" /> Get WML owner and last-modified details during wml list
  • <input type="checkbox" disabled="" /> Allow custom operators
  • <input type="checkbox" disabled="" /> Add defaults to CLI --help commands
  • <input type="checkbox" disabled="" /> Strategy for Python Opjects (e.g. callables) - maybe import statement?
  • <input type="checkbox" disabled="" /> Backport existing Python DAGs to WMLs
  • <input type="checkbox" disabled="" /> Allow YML updates to propogate to WMLs
  • <input type="checkbox" disabled="" /> Allow user specified Airflow Version - isolate to docker or something so we can run 2.7 if we want?

MVP Usage Pattern

To run as a user:

  1. Install with pip install airflow-windmill
  2. Run windmill init to create a local Windmill project
  3. Run windmill run from this folder to run the app locally

To run as a dev:

  1. Clone from git
  2. Run poetry install
  3. ?Run windmill-dev install-node-depts
  4. Run windmill-dev start to start a flask server and use parcel to serve frontend

Future Usage Patterns

  • Auto-sync for windmill project to git

Getting Started

This package can be installed and run using Pip:

pip install windmill-airflow

Deployment

# Run NPM build
cd windmill/http/app/
npm run-script build

# Update git ...
cd ../../../

# Poetry build
poetry build
poetry publish

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for windmill-airflow, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size windmill_airflow-0.0.2-py3-none-any.whl (1.5 MB) File type Wheel Python version py3 Upload date Hashes View
Filename, size windmill-airflow-0.0.2.tar.gz (1.5 MB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page