airflow-nomad 0.1.0

Nomad operators and configuration for long-running tasks

airflow-nomad

Run and monitor Nomad-managed jobs from Apache Airflow.

from airflow import DAG
from airflow_nomad import Job, Nomad, NomadAirflowConfiguration, Task, TaskGroup

dag = DAG(dag_id="nightly-nomad", schedule="@daily")
config = NomadAirflowConfiguration(
    job=Job(
        id="nightly",
        type="batch",
        task_groups=[
            TaskGroup(
                name="nightly",
                tasks=[Task(name="nightly", driver="exec", config={"command": "/opt/jobs/nightly"})],
            )
        ],
    )
)
Nomad(dag=dag, cfg=config)

The generated task lifecycle writes and registers the jobspec, monitors current allocations with airflow-ha, handles retriggers, stops the job, and optionally removes the generated configuration.

Documentation

• Tutorial: run a Nomad job from Airflow
• How-to guides
• Why Airflow owns the lifecycle
• API reference

Published documentation is available at airflow-laminar.github.io/airflow-nomad.

Ecosystem

• nomad-pydantic supplies jobspec models and the Nomad CLI client.
• supervisor-pydantic, systemd-pydantic, and cron-pydantic model alternative runtimes.
• airflow-supervisor and airflow-systemd provide analogous long-running job lifecycles.
• airflow-cron converts cron jobs into ordinary Airflow tasks.
• airflow-pydantic supplies declarative task and connection models.
• airflow-config produces YAML-driven DAGs.

[!NOTE] This library was generated using copier from the Base Python Project Template repository.