Overview

bisos.airflow is a BISOS package for managing an Apache Airflow platform — webserver (API server), scheduler, triggerer, metadata database, and DAG development/testing — as a BISOS Capability Bundle (CBS/CBM) of systemd-planted Command Services.

With CBM (Capability Bundle Materialization), Airflow’s four processes are installed as independent host systemd units rather than one monolithic airflow standalone process: a planted pointer leaf (cbmProc.spcs) subprocess-invokes a Capability Bundle Specification (airflow-cbs.pcs) which declares package installation (sbom), the four systemd units, and the assembly step needed to bring up a working airflow.here installation on a single host. This gives independent start/stop/status and independent logs per role, without the operational overhead of a distributed executor (Celery/Kubernetes) or containers.

Package Documentation At Github

The information below is a subset of the full of documentation for this bisos-pip package. More complete documentation is available at: https://github.com/bisos-pip/airflow-cs

Capability Materialization

The figure below shows the top-down layering of airflow.here’s Capability Materialization: the planted CBM pointer leaf (/bisos/platform/sys/cbm/collective/hereWeb/airflow/cbmProc.spcs) points to (and does not duplicate) the CBS (py3/bin/airflow-cbs.pcs), which declares the bundle of airflow-sbom.pcs, the 4 airflow-*-sysd.pcs units, and airflow-assemble.cs. py3/bin/airflowAdmin.cs sits below all of that as the day-to-day admin-facing CS an operator runs against the materialized installation.

Regenerate this figure (after editing py3/images/airflow-graphviz.pcs):

cd py3/images
./airflow-graphviz.pcs --format=all -i ngProcess all

Table of Contents TOC

• Overview

• Package Documentation At Github

• Capability Materialization

• The 4 systemd Units and DB Init

• Installation

  • Installation With pip

  • Installation With pipx

• Usage

  • Day-to-Day Admin: ``airflowAdmin.cs` <#day-to-day-admin-airflowadmincs>`__

  • Direct Airflow CLI Cheat Sheet: ``quickAirflow.cs` <#direct-airflow-cli-cheat-sheet-quickairflowcs>`__

• Key Files

• Part of BISOS — ByStar Internet Services Operating System

• Documentation and Blee-Panels

• Support

The 4 systemd Units and DB Init

Four planted *-sysd.pcs units live in py3/bin/, all delegating to the same airflow CLI executable, each running a different sub-command:

airflow-db-sysd.pcs is a oneshot unit with RemainAfterExit=yes: it runs airflow db migrate once to bring the metadata database schema up to date, then reports itself “active” without staying resident. The three long-running units each declare Requires=/=After=airflow-db.service, so systemd guarantees the migration has been attempted before any of the three service processes starts.

All four units, together with package installation, are declared by py3/bin/airflow-cbs.pcs and materialized at this host via py3/bin/cbmProc-airflow.spcs.

Installation

You can install bisos.airflow with pip or pipx.

pip install bisos.airflow

pipx install bisos.airflow

Usage

airflowAdmin.cs -i hostInfo       # fqdn, port, AIRFLOW_HOME, DAG dir, log dir
airflowAdmin.cs -i servicesInfo   # links for operating the 4 systemd units
airflowAdmin.cs -i usersInfo      # list/create users; default admin/password
airflowAdmin.cs -i fullUpdate     # all three together

quickAirflow.cs

Key Files

An overview of the relevant files of the bisos.airflow package (starting-point; to be expanded as the package develops):

• py3/bin/airflowAdmin.cs — day-to-day admin CS (hostInfo, servicesInfo, usersInfo, fullUpdate).

• py3/bin/quickAirflow.cs — direct-command cheat sheet for the airflow CLI and systemd units.

• py3/bin/airflow-cbs.pcs — CBS declaring the bundle (sbom + 4 sysd units + assemble).

• py3/bin/cbmProc-airflow.spcs — planted CBM pointer leaf that materializes the CBS.

• py3/bin/airflow-sbom.pcs — package installation via bisos.sbom.

• py3/bin/airflow-db-sysd.pcs — oneshot DB migration unit.

• py3/bin/airflow-webserver-sysd.pcs, airflow-scheduler-sysd.pcs, airflow-triggerer-sysd.pcs — the three long-running units.

• py3/bin/airflow-assemble.cs — assembly/glue step.

• py3/bin/airflow-here-dns.pcs — .here fake-domain DNS registration for airflow.here.

• py3/images/airflow-graphviz.pcs — source for the Capability Materialization figure.

• py3/tests/verify.sh — smoke test for airflowAdmin.cs and quickAirflow.cs.

• py3/setup.py, py3/pypiProc.sh — PyPI packaging (setup.py is dblock-driven; do not hand-edit).

Part of BISOS — ByStar Internet Services Operating System

Layered on top of Debian, BISOS (By* Internet Services Operating System) is a unified and universal framework for developing both internet services and software-service continuums that use internet services. See Bootstrapping ByStar, BISOS and Blee for information about getting started with BISOS.

BISOS is a foundation for The Libre-Halaal ByStar Digital Ecosystem which is described as a cure for losses of autonomy and privacy in a book titled: Nature of Polyexistentials

bisos.airflow is part of BISOS. It is a standalone package that can be used independently of the full BISOS environment.

Documentation and Blee-Panels

bisos.airflow is part of the ByStar Digital Ecosystem http://www.by-star.net.

This module’s primary documentation is in the form of Blee-Panels. Blee-Panels are in the ./panels directory. From within Blee and BISOS these panels are accessible under the Blee “Panels” menu.

See file:./panels/bisos.airflow/_nodeBase_/fullUsagePanel-en.org for a starting point.

bisos.airflow is best developed with Blee, the By* BISOS Libre-Halaal Emacs Environment — a layer on top of Emacs and BISOS which creates a comprehensive integrated usage and development environment.

Support