Overview

bisos.airflow is a BISOS package for managing an Apache Airflow platform — webserver (API server), scheduler, triggerer, metadata database, and DAG development/testing — as a BISOS Capability Bundle (CBS/CBM) of systemd-planted Command Services.

With CBM (Capability Bundle Materialization), Airflow’s five roles are installed as independent host systemd units rather than one monolithic airflow standalone process: a planted pointer leaf (cbmProc.spcs) subprocess-invokes a Capability Bundle Specification (airflow-cbs.pcs) which declares package installation (sbom), the five systemd units, and the assembly step needed to bring up a working airflow.here installation on a single host. This gives independent start/stop/status and independent logs per role, without the operational overhead of a distributed executor (Celery/Kubernetes) or containers.

Airflow runs as a dedicated airflow system account, not root, with AIRFLOW_HOME=/opt/airflow and a PostgreSQL metadata DB, and it targets Airflow 3. The full description is in the airflowUsage Blee panel.

Package Documentation At Github

The information below is a subset of the full of documentation for this bisos-pip package. More complete documentation is available at: https://github.com/bisos-pip/airflow-cs

Capability Materialization

The figure below shows the top-down layering of airflow.here’s Capability Materialization: the planted CBM pointer leaf (/bisos/platform/sys/cbm/collective/hereWeb/airflow/cbmProc.spcs) points to (and does not duplicate) the CBS (py3/bin/airflow-cbs.pcs), which declares the bundle of airflow-sbom.pcs, the 5 airflow-*-sysd.pcs units, and airflow-assemble.cs. py3/bin/airflowAdmin.cs sits below all of that as the day-to-day admin-facing CS an operator runs against the materialized installation.

Regenerate this figure (after editing py3/images/airflow-graphviz.pcs):

cd py3/images
./airflow-graphviz.pcs --format=all -i ngProcess all

Table of Contents TOC

• Overview

• Package Documentation At Github

• Capability Materialization

• The 5 systemd Units and DB Init

• The Library: bisos.airflow.airflowInfo

• Installation

  • Installation With pip

  • Installation With pipx

• Usage

  • Day-to-Day Admin: ``airflowAdmin.cs` <#day-to-day-admin-airflowadmincs>`__

  • Direct Airflow CLI Cheat Sheet: ``quickAirflow.cs` <#direct-airflow-cli-cheat-sheet-quickairflowcs>`__

• Key Files

• Part of BISOS — ByStar Internet Services Operating System

• Documentation and Blee-Panels

• Support

The 5 systemd Units and DB Init

Five planted *-sysd.pcs units live in py3/bin/, all delegating to the same airflow CLI executable, each running a different sub-command. All five run as User=airflow and read EnvironmentFile=/etc/default/airflow, which sets AIRFLOW_HOME=/opt/airflow, the PostgreSQL connection string and the Airflow 3 API URLs. airflow-sbom.pcs creates the account, the directories, that file and the metadata DB.

airflow-db-sysd.pcs is a oneshot unit with RemainAfterExit=yes: once PostgreSQL is up, it runs airflow db migrate to bring the metadata schema up to date, then reports itself “active” without staying resident. The four long-running units each declare Requires=/=After=airflow-db.service, so systemd guarantees the migration has been attempted before any of them starts. airflow-dag-processor is required on Airflow 3, which parses DAG files in a standalone process; without it no DAG is ever registered.

All five units, together with package installation, are declared by py3/bin/airflow-cbs.pcs and materialized on this host by the CBM leaf /bisos/platform/sys/cbm/collective/hereWeb/airflow/cbmProc.spcs (the package ships the same pointer as py3/bin/cbmProc-airflow.spcs).

The Library: bisos.airflow.airflowInfo

The facts of the installation used to be repeated across the sbom, airflowAdmin.cs, the five units, the DNS and vhost files and the generated /etc/default/airflow, with nothing making them agree. They now have one source, py3/bisos/airflow/airflowInfo.py. It is pure: importing it has no side effects and touches no host state.

Pure generators sit next to the constants: envFileContent() is the text /etc/default/airflow is generated with, and sysdServiceLines() the User, Group and EnvironmentFile lines of every unit. envFileDrift() and unitDrift() compare the live files with them. airflowAdmin.cs -i driftCheck runs that comparison read-only.

/etc/default/airflow is still written only when absent, so hand overrides survive. A lack of a generated line in a live file is therefore reported as drift, never fixed.

The sbom, the five units, airflow-here-dns.pcs, airflow-wvd.pcs, airflowAdmin.cs and quickAirflow.cs all read airflowInfo. The .pcs files import bisos.airflow, so the package must be installed (see Installation).

Installation

You can install bisos.airflow with pip or pipx.

pip install bisos.airflow

pypiProc.sh -i pkgReInstall edit /bisos/venv/bisos3

pipx install bisos.airflow

Usage

airflowAdmin.cs -i hostInfo       # fqdn, port, AIRFLOW_HOME, DAG dir, log dir
airflowAdmin.cs -i servicesInfo   # links for operating the systemd units
airflowAdmin.cs -i dagsInfo       # list, trigger, pause, graph, parsing health
airflowAdmin.cs -i usersInfo      # list/create users; default admin/password
airflowAdmin.cs -i fullUpdate     # all of the above
airflowAdmin.cs -i driftCheck     # live env file and units against airflowInfo; read-only

quickAirflow.cs

Key Files

An overview of the relevant files of the bisos.airflow package (starting-point; to be expanded as the package develops):

• py3/bisos/airflow/airflowInfo.py — the single source of the installation’s facts, the env-file and unit generators, and the drift functions.

• py3/bin/airflowAdmin.cs — day-to-day admin CS (hostInfo, servicesInfo, dagsInfo, usersInfo, fullUpdate, driftCheck).

• py3/bin/quickAirflow.cs — direct-command cheat sheet for the airflow CLI and systemd units.

• py3/bin/airflow-cbs.pcs — CBS declaring the bundle (sbom + 5 sysd units + assemble).

• py3/bin/cbmProc-airflow.spcs — planted CBM pointer leaf that materializes the CBS.

• py3/bin/airflow-sbom.pcs — package installation via bisos.sbom, the airflow account and the PostgreSQL metadata DB.

• py3/bin/airflow-db-sysd.pcs — oneshot DB migration unit.

• py3/bin/airflow-webserver-sysd.pcs, airflow-scheduler-sysd.pcs, airflow-triggerer-sysd.pcs, airflow-dag-processor-sysd.pcs — the four long-running units.

• py3/bin/airflow-assemble.cs — assembly/glue step (DNS + nginx vhost).

• py3/bin/airflow-here-dns.pcs — .here fake-domain DNS registration for airflow.here.

• py3/bin/airflow-wvd.pcs — nginx virtual host for airflow.here.

• py3/images/airflow-graphviz.pcs — source for the Capability Materialization figure.

• py3/tests/verify.sh — smoke test for airflowAdmin.cs, quickAirflow.cs and airflowInfo.

• py3/setup.py, py3/pypiProc.sh — PyPI packaging (setup.py is dblock-driven; do not hand-edit).

Part of BISOS — ByStar Internet Services Operating System

Layered on top of Debian, BISOS (By* Internet Services Operating System) is a unified and universal framework for developing both internet services and software-service continuums that use internet services. See Bootstrapping ByStar, BISOS and Blee for information about getting started with BISOS.

BISOS is a foundation for The Libre-Halaal ByStar Digital Ecosystem which is described as a cure for losses of autonomy and privacy in a book titled: Nature of Polyexistentials

bisos.airflow is part of BISOS. It is a standalone package that can be used independently of the full BISOS environment.

Documentation and Blee-Panels

bisos.airflow is part of the ByStar Digital Ecosystem http://www.by-star.net.

This module’s primary documentation is in the form of Blee-Panels. Blee-Panels are in the ./panels directory. From within Blee and BISOS these panels are accessible under the Blee “Panels” menu.

See file:./panels/bisos.airflow/_nodeBase_/fullUsagePanel-en.org for a starting point, and airflowUsage for the full description of what the package does.

bisos.airflow is best developed with Blee, the By* BISOS Libre-Halaal Emacs Environment — a layer on top of Emacs and BISOS which creates a comprehensive integrated usage and development environment.

Support