airflow-nomad

Run and monitor Nomad-managed jobs from Apache Airflow.

from datetime import datetime, timezone

from airflow import DAG
from airflow_nomad import Job, Nomad, NomadAirflowConfiguration, Task, TaskGroup

dag = DAG(
    dag_id="nightly-nomad",
    schedule="@daily",
    start_date=datetime(2025, 1, 1, tzinfo=timezone.utc),
    catchup=False,
)
config = NomadAirflowConfiguration(
    working_dir="/var/tmp/nightly-nomad",
    job=Job(
        id="airflow-nightly",
        type="batch",
        namespace="default",
        datacenters=["dc1"],
        task_groups=[
            TaskGroup(
                name="nightly",
                tasks=[Task(name="nightly", driver="exec", config={"command": "/bin/sleep", "args": ["5"]})],
            )
        ],
    )
)
Nomad(dag=dag, cfg=config)

The generated task lifecycle writes and registers the jobspec, monitors current allocations with airflow-ha, handles retriggers, stops the job, and optionally removes the generated configuration on successful completion. Workers use the Nomad CLI to reach local or remote clusters through their HTTP API. Define the same lifecycle in inline Python or airflow-config YAML.

Documentation

Start with the tutorial to run a short batch job on a development cluster and check cleanup. For an existing cluster, follow the Python or airflow-config guide. Both use the same cluster credentials and namespace settings.

Use the observability guide to forward task logs, attach failure callbacks, or monitor a persistent job between management runs.

The API reference lists configuration defaults, monitoring settings, and task boundaries. Why Airflow owns the lifecycle explains allocation state, persistent services, and scheduling ownership.

Published documentation is available at airflow-laminar.github.io/airflow-nomad.

Ecosystem

• nomad-pydantic supplies jobspec models and the Nomad CLI client.
• supervisor-pydantic, systemd-pydantic, and cron-pydantic model alternative runtimes.
• airflow-supervisor and airflow-systemd provide analogous long-running job lifecycles.
• airflow-cron converts cron jobs into ordinary Airflow tasks.
• airflow-pydantic supplies declarative task and connection models.
• airflow-config produces YAML-driven DAGs.