Apache Airflow DagSyncer

Category	Badges
License
PyPI

apache-airflow-dagsyncer ships serialized dags from a data plane to an Apache Airflow control plane over HTTP/S - edge-executor style. The control plane never parses dag files and never dials into the data plane.

  Data plane (remote)                     Control plane
+------------------------+   HTTP/S    +---------------------------+
| DagProcessor (run-once |  -------->  | Airflow api-server        |
|   against local SQLite)|  manifest   | +-----------------------+ |
| syncer (push)          |             | | dagsyncer plugin      | |
|   extracts serialized  |             | |  auth + validation    | |
|   dags, POSTs manifest |             | |  dag parsing update   | |
+------------------------+             | |  -> control-plane DB  | |
                                       | +-----------------------+ |
                                       +---------------------------+

On each deployment the syncer runs the DagProcessor once against a local SQLite database, extracts the serialized dag JSON, and pushes a full bundle manifest to the listener. The listener is an Airflow plugin mounted into the api-server (edge3 provider pattern), so it inherits the api-server's TLS and deployment. Ingestion goes through Airflow's own dag parsing update path, so versioning, hashing, and consistency stay Airflow's responsibility. Dags absent from the manifest are deactivated.

See PROTOCOL.md for the wire protocol.

Status: early development / AIP-92 workaround. Push client and listener plugin work end-to-end; full scheduler/edge-worker loop validation is in progress. This package exists because AIP-92 - upstream's planned DAG-processor isolation - is still a draft and not yet implemented end-to-end. When AIP-92 lands natively, dagsyncer should become unnecessary and can be deprecated.

Table of contents

• Requirements
• Installing from PyPI
• Usage
• Contributing
• License

Requirements

The syncer side has no runtime dependencies. The listener side requires Apache Airflow, installed via the [listen] extra:

pip install "apache-airflow-dagsyncer[listen]"

Installing from PyPI

pip install apache-airflow-dagsyncer

Or with uv:

uv pip install apache-airflow-dagsyncer

Usage

Control plane - install the listener next to the api-server and configure the shared secret; the plugin is discovered automatically:

pip install "apache-airflow-dagsyncer[listen]"
export AIRFLOW__DAGSYNCER__API_SECRET_KEY="..."
airflow api-server

Data plane - parse a bundle and push it to the control plane:

apache-airflow-dagsyncer push \
  --bundle-path /opt/dags \
  --bundle-name my-dags \
  --listener-url https://airflow.example.com \
  --token "$DAGSYNCER_TOKEN"

Contributing

Want to help build this project? Set up a development environment with uv:

git clone https://github.com/abhishekbhakat/apache-airflow-dagsyncer.git
cd apache-airflow-dagsyncer
uv sync
uv run apache-airflow-dagsyncer --help

License

Apache License, Version 2.0