apache-airflow-dagsyncer 0.0.1rc2

CLI utility to sync DAG files into an Apache Airflow dags folder

Apache Airflow DagSyncer

Category	Badges
License
PyPI

apache-airflow-dagsyncer ships serialized dags from a data plane to an Apache Airflow control plane over HTTP/S - edge-executor style. The control plane never parses dag files and never dials into the data plane.

  Data plane (remote)                        Control plane
+------------------------+    HTTP/S      +------------------------+
| DagProcessor (run-once |  ---------->   | listener               |
|   against local SQLite)|   manifest     |   auth + validation    |
| syncer (push)          |                |   SerializedDagModel   |
|   extracts serialized  |                |     .write_dag()       |
|   dags, POSTs manifest |                |   -> control-plane DB  |
+------------------------+                +------------------------+

On each deployment the syncer runs the DagProcessor once against a local SQLite database, extracts the serialized dag JSON, and pushes a full bundle manifest to the listener. The listener ingests it through Airflow's own update path, so versioning, hashing, and consistency stay Airflow's responsibility. Dags absent from the manifest are deactivated.

See PROTOCOL.md for the wire protocol.

Status: early development. The push and listen commands are not implemented yet; the wire protocol (v1) is defined.

Table of contents

• Requirements
• Installing from PyPI
• Usage
• Contributing
• License

Requirements

Component	Version
Python	3.12+

The syncer side has no runtime dependencies. The listener side requires Apache Airflow, installed via the [listen] extra:

pip install "apache-airflow-dagsyncer[listen]"

Installing from PyPI

pip install apache-airflow-dagsyncer

Or with uv:

uv pip install apache-airflow-dagsyncer

Usage

Data plane - parse a bundle and push it to the control plane:

apache-airflow-dagsyncer push \
  --bundle-path /opt/dags \
  --bundle-name my-dags \
  --listener-url https://cp.example.com:8793 \
  --token "$DAGSYNCER_TOKEN"

Control plane - run the listener:

apache-airflow-dagsyncer listen --host 0.0.0.0 --port 8793

Contributing

Want to help build this project? Set up a development environment with uv:

git clone https://github.com/abhishekbhakat/apache-airflow-dagsyncer.git
cd apache-airflow-dagsyncer
uv sync
uv run apache-airflow-dagsyncer --help

License

Apache License, Version 2.0