airflow-provider-inbound-webhooks 0.1.0

Turn incoming webhooks into asset events that trigger your Airflow Dags.

airflow-provider-inbound-webhooks lets any service that can send a signed or secret-carrying webhook trigger your Apache Airflow Dags.

Each verified delivery becomes an asset event, named after what happened. A Dag subscribes by scheduling on that name, and Airflow does the rest. There is no routing code to write, and adding a sender is one Airflow connection.

GitHub, GitLab and Jira are built in, with more services on the way. Any other service that signs its deliveries with HMAC-SHA256 or sends a secret header works today, described in its connection.

@dag(schedule=Asset("demo/billing/deployed"))
def smoke_test_billing(): ...

Why

Airflow can be triggered from outside through its REST API, but webhook senders can't use it: they cannot obtain or attach an Airflow API token. They sign their requests with a shared secret instead, and each service posts its own payload format. The usual workarounds are a relay service in front of Airflow, or sensors that poll the other service and hold worker or triggerer capacity while they wait.

This plugin adds the missing piece: an endpoint on the Airflow API server that verifies each service's signature and records the delivery as an asset event.

Quick start

Try it with the bundled test sender, so no third-party account is needed. Run these commands on the machine running the Airflow API server (Airflow 3.1.1 or later).

1. Install the plugin:

   pip install airflow-provider-inbound-webhooks "apache-airflow==$(airflow version)"
   

   Pinning apache-airflow to the version you run keeps pip from changing it.

2. Enable the webhook name demo, then restart the API server. The plugin serves only the names listed in webhook_names, so add this to airflow.cfg in your Airflow home:

   [inbound_webhooks]
   webhook_names = demo
   

   Or set AIRFLOW__INBOUND_WEBHOOKS__WEBHOOK_NAMES=demo in the API server's environment. After the restart, airflow plugins lists webhooks.

3. Create the webhook. It is a connection, named webhook_demo for the URL /webhooks/demo:

   airflow connections add webhook_demo --conn-type inbound_webhook --conn-password 'change-me' \
     --conn-extra '{"source_type": "hmac_json", "signature_header": "X-Signature",
                    "event_field": "event", "path_field": "service"}'
   

   Or in the Airflow UI

   Under Admin → Connections, add a connection of type Inbound Webhook:

   Field	Value
   Connection Id	webhook_demo
   Shared secret	change-me
   Source type	hmac_json
   Signature header	X-Signature
   Event field	event
   Path field	service

4. Add a Dag that runs on the event, and unpause it, since new Dags start paused:

   dags_folder="$(airflow config get-value core dags_folder)"
   mkdir -p "$dags_folder"
   cat > "$dags_folder/quickstart.py" <<'EOF'
   """Quick start Dag: runs when the ``demo`` webhook receives a ``deployed`` event for the ``billing`` service."""
   
   from __future__ import annotations
   
   from airflow.sdk import Asset, dag, task
   
   billing_deployed = Asset("demo/billing/deployed")
   
   
   @dag(schedule=billing_deployed)
   def smoke_test_billing():
       @task
       def smoke_test(*, triggering_asset_events=None):
           # One run can carry several deliveries (always possible before Airflow 3.4).
           for event in triggering_asset_events[billing_deployed]:
               print(f"Smoke-testing billing after deployment: {event.extra['payload']}")
   
       smoke_test()
   
   
   smoke_test_billing()
   EOF
   airflow dags reserialize  # parse it now rather than at the next scan of the Dags folder
   airflow dags unpause smoke_test_billing
   

5. Send a signed test delivery with the test sender the package includes:

   python -m airflow_inbound_webhooks.send_test_event \
     --url http://localhost:8080/webhooks/demo --secret 'change-me' \
     --data '{"event": "deployed", "service": "billing"}'
   

   It prints the response, and smoke_test_billing starts a run:

   HTTP 202
   {"status":"accepted","asset_events":{"demo/billing/deployed":1},"candidates":["demo/billing/deployed","demo/*/deployed"]}
   

   Anything else? See Responses and troubleshooting.

A webhook's connection is read again within a minute of a change, so changing its secret or options needs no restart. Adding a webhook does, because its name must be added to webhook_names.

Connect a service

Every service takes the same steps: in Airflow, enable the webhook's name and create a connection holding a secret; in the service, add a webhook with that secret; then, in Airflow, schedule a Dag on one of its events.

More built-in services are on the way. To ask for one, open a feature request with a link to the service's webhook documentation.

Before you start:

• The plugin is installed on the API server: Quick start, step 1.
• The service can reach the API server. A cloud service, such as GitHub.com, GitLab.com or Jira Cloud, needs a public HTTPS address: put a reverse proxy in front of the API server that forwards only the paths of your webhooks, such as /webhooks/github (see Security). <your-airflow> below stands for that address.

Connect GitHub

In Airflow

1. Add github to webhook_names, the comma-separated list in airflow.cfg, and restart the API server.

2. Create the connection webhook_github with a new secret, and copy the secret it prints:

   SECRET="$(openssl rand -hex 32)" && echo "$SECRET"
   airflow connections add webhook_github --conn-type inbound_webhook --conn-password "$SECRET"
   

In GitHub

3. In the repository, open Settings → Webhooks → Add webhook. To cover every repository of an organization, use the organization's Settings → Webhooks instead, as an organization owner.

4. Fill in the form, then select Add webhook:

   Field	Value
   Payload URL	https://<your-airflow>/webhooks/github
   Content type	application/json. The default, application/x-www-form-urlencoded, is refused.
   Secret	The secret from step 2
   Which events would you like to trigger this webhook?	Let me select individual events, then the ones your Dags need, such as Pull requests or Pushes
   Active	Checked

5. Check it. GitHub sends a ping right away: on the webhook's Recent Deliveries tab it should have a green check, and its response should be 202 with "status":"no_subscribers".

In Airflow

6. Add a Dag that schedules on an event, such as Asset("github/my-org/web/pull_request/labeled/deploy-preview"), and unpause it. To find a name, trigger the event once and copy one of the candidates from its response under Recent Deliveries, or see Event names.

Connect GitLab

In Airflow

1. Add gitlab to webhook_names, the comma-separated list in airflow.cfg, and restart the API server.

2. Create the connection webhook_gitlab with a new secret, and copy the secret it prints:

   SECRET="$(openssl rand -hex 32)" && echo "$SECRET"
   airflow connections add webhook_gitlab --conn-type inbound_webhook --conn-password "$SECRET"
   

In GitLab

3. In the project, open Settings → Webhooks → Add new webhook. To cover every project of a group, use the group's Settings → Webhooks instead, as a group owner (GitLab Premium).

4. Fill in the form, then select Add webhook:

   Field	Value
   URL	https://<your-airflow>/webhooks/gitlab
   Secret token	The secret from step 2. GitLab also offers a Signing token, which this plugin does not check yet; you can set both.
   Trigger	The events your Dags need, such as Push events, Merge request events or Pipeline events
   Enable SSL verification	Checked

5. Check it. In the list of webhooks, select Test → Push events. GitLab should report HTTP 202; Edit → Recent events → View details shows the response.

In Airflow

6. Add a Dag that schedules on an event, such as Asset("gitlab/platform/api/pipeline/success"), and unpause it. To find a name, trigger the event once and copy one of the candidates from its response under Recent events, or see Event names.

Connect Jira

In Airflow

1. Add jira to webhook_names, the comma-separated list in airflow.cfg, and restart the API server.

2. Create the connection webhook_jira with a new secret, and copy the secret it prints:

   SECRET="$(openssl rand -hex 32)" && echo "$SECRET"
   airflow connections add webhook_jira --conn-type inbound_webhook --conn-password "$SECRET"
   

In Jira, as a Jira administrator

3. Open Settings (⚙) → System → WebHooks → Create a WebHook. On Jira Data Center, the menu is Administration (⚙) → System → WebHooks.

4. Fill in the form, then create the webhook:

   Field	Value
   Name	Any, such as Airflow
   Status	Enabled
   URL	https://<your-airflow>/webhooks/jira
   Secret	The secret from step 2. Jira cannot show it again once saved.
   Issue related events	A JQL filter such as project = OPS, and the events your Dags need, such as issue created and updated
   Exclude body	Unchecked

In Airflow

5. Add a Dag that schedules on an event, such as Asset("jira/OPS/issue_updated/status/Ready for Deploy"), and unpause it. See jira for the names Jira events get.

6. Check it by making that change in Jira, here moving an OPS issue to Ready for Deploy: the Dag starts a run. Jira does not show webhook responses, so if nothing happens, check the API server log: it reports a delivery whose secret does not match, or a misconfigured connection. Otherwise check that jira is in webhook_names, that Exclude body is unchecked and that the Dag is unpaused. With [logging] logging_level = DEBUG on the API server, the log also lists the names a delivery offered when no Dag schedules on any of them.

Connect any other service

For a service without a built-in source type, such as a deploy tool named deploybot here, you describe its deliveries in the connection.

In the service's webhook documentation, find:

In Airflow

1. Add deploybot to webhook_names, the comma-separated list in airflow.cfg, and restart the API server.

2. Create the connection webhook_deploybot with a new secret and those options, and copy the secret it prints:

   SECRET="$(openssl rand -hex 32)" && echo "$SECRET"
   airflow connections add webhook_deploybot --conn-type inbound_webhook --conn-password "$SECRET" \
     --conn-extra '{"source_type": "hmac_json", "signature_header": "X-Signature",
                    "signature_prefix": "sha256=", "event_field": "type", "path_field": "service"}'
   

   hmac_json and token_header list every option, and the Inbound Webhook connection form in the Airflow UI has a field for each.

In the service

3. Add a webhook with the URL https://<your-airflow>/webhooks/deploybot and the secret from step 2, sending JSON.

In Airflow

4. Add a Dag that schedules on an event, such as Asset("deploybot/billing/deployed"), and unpause it. To find a name, send the event once and copy one of the candidates from the response, or see Event names.

A service whose signature covers more than the body, such as a timestamp, or that needs a handshake when the webhook is registered, needs a built-in source type: see Other services.

How it works

See the diagram of this flow on GitHub.

The plugin never runs Dags itself. It records asset events, Airflow queues every Dag that schedules on them, and the scheduler creates the runs. So one delivery can start several Dags, paused Dags stay paused, and asset expressions such as a | b work as usual.

Event names

Every delivery is named <webhook name>/<path>/<event>/<action>/<detail>, and every prefix of that name can be subscribed to. Each prefix is also offered with * in place of each trailing part of the path, so * stands for anything below that point. For the quick start delivery the names are:

demo/billing/deployed     billing deployments
demo/*/deployed           deployments of any service

When a label is added to a pull request, the built-in GitHub source offers nine names, from broad to specific:

github/my-org/web/pull_request                           any pull request activity in my-org/web
github/my-org/*/pull_request                             ... in any repository of my-org
github/*/pull_request                                    ... in any repository
github/my-org/web/pull_request/labeled                   any label added
github/my-org/*/pull_request/labeled
github/*/pull_request/labeled
github/my-org/web/pull_request/labeled/deploy-preview    the deploy-preview label added
github/my-org/*/pull_request/labeled/deploy-preview
github/*/pull_request/labeled/deploy-preview

Deeper paths work the same way. For a GitLab project platform/backend/api, gitlab/platform/backend/*/... covers the backend subgroup, gitlab/platform/*/... the whole platform group, subgroups included, and gitlab/*/... every project.

Finding a name: every response lists the names it offered as candidates, so send the event once and copy one. A delivery that matches no subscribed name is answered with 202 no_subscribers and nothing is stored.

Non-ASCII characters are folded (🚀 deploy → deploy) on every database, because MySQL stores asset names as latin1 and fails any lookup that includes a non-ASCII name. Parts that end up empty are dropped, so no name contains //: a path ops/🚀/api becomes ops/api, and a label 🚀 adds no detail.

Stability: event names are a public interface (see Versioning), and the one most easily broken, because a Dag whose name stops matching does not fail; it just stops running. Changing a name that an existing delivery produces, for example how it is normalized or what its detail is, is a breaking change, and the changelog lists the old and new names under "Changed event names". Adding names, such as a detail for an event that had none, is not, because the shorter names are still offered.

Source types

The connection's source_type decides how deliveries are verified and named. hmac_json and token_header work with any service and are configured through connection options; the others are built in for one service each. It defaults to the webhook name, so webhook_github, webhook_gitlab and webhook_jira need nothing but a password.

Each asset event stores the delivery's body as extra["payload"], as received, unless every delivery also carries a full copy of the object it is about. Of the built-in services only Jira does, sending the whole issue each time, so jira stores a summary by default.

hmac_json: services that sign the body

For services that sign the raw body with HMAC-SHA256 and send the digest in a header. Prefer this over token_header whenever a service can sign, because the signature also proves the body was not changed.

token_header: services that send a secret header

For services that can only send the shared secret in a header you configure. A bare token proves who sent a delivery but not that it was unmodified, so it relies on TLS.

github

GitHub.com and GitHub Enterprise Server. An organization webhook covers every repository in the organization.

Names use GitHub's own event and action names. The detail is the label for labeled and unlabeled, the branch for push (tag pushes have none), and the workflow name for workflow_run. A second GitHub instance is a second connection, e.g. webhook_ghes with extra {"source_type": "github"}.

gitlab

GitLab.com and self-managed GitLab. Group webhooks cover every project in a group (a GitLab Premium feature); otherwise add the webhook per project. To subscribe to every project in a group, subgroups included, use gitlab/<group>/*/.... GitLab sends the secret token itself rather than signing the body.

Names use the payload's object_kind, then object_attributes.action (or its status, for pipelines). The detail is the branch for push and the tag for tag_push. GitLab has no label events, so a label added to a merge request or issue is reported as <object_kind>/labeled/<label>.

jira

Jira Cloud and Jira Data Center. Events are named from the body, so the webhook must not exclude it. Webhooks that Connect or OAuth 2.0 apps register are authenticated differently and not supported.

Names use the payload's webhookEvent without its jira: prefix, under the issue's project key. The detail is the issue type for issue_created. issue_updated offers a name for each field in the changelog, with the new status for status changes and, as for GitLab, issue_updated/labeled/<label> for each label added. Moving an issue of project OPS to Ready for Deploy and adding the label hotfix offers:

jira/OPS/issue_updated                            any change to an OPS issue
jira/*/issue_updated                              ... in any project
jira/OPS/issue_updated/status                     any transition
jira/*/issue_updated/status
jira/OPS/issue_updated/status/Ready for Deploy    moved to Ready for Deploy
jira/*/issue_updated/status/Ready for Deploy
jira/OPS/issue_updated/labels                     any change to the labels
jira/*/issue_updated/labels
jira/OPS/issue_updated/labeled                    any label added
jira/*/issue_updated/labeled
jira/OPS/issue_updated/labeled/hotfix             the hotfix label added
jira/*/issue_updated/labeled/hotfix

Field names are the changelog's, such as Story Points for a custom field. Events about neither an issue nor a project, such as sprint_started, have no path: jira/sprint_started.

Jira sends the whole issue, every custom field included, with each delivery. So each asset event stores a summary: the payload with the issue's fields reduced to project, issuetype, status, summary, labels, priority, assignee, resolution, fixVersions and parent. The changelog and the rest of the payload are kept. A Dag that needs other fields can fetch the issue from Jira, which also gives its current state, or the connection can store the whole payload:

Jira sends issue_updated for every edit, so select only the events your Dags need and narrow the webhook with a JQL filter such as project = OPS. Jira Cloud retries a delivery that fails or times out, so a Dag may occasionally see one twice. Its delivery_id, Jira Cloud's X-Atlassian-Webhook-Identifier, stays the same across retries.

On a Jira without webhook secrets, an Automation for Jira rule with a "Send web request" action can send the secret in a header for token_header instead.

Other services

Built-in source types for more services are on the way. A service needs one when its signature covers more than the body (for example a timestamp plus the body), when it needs a verification handshake as the webhook is registered, or to name its events in more detail than one field.

To ask for a service, open a feature request with a link to its webhook documentation. To add one yourself: a source type is a small class with verify() and parse(); see CONTRIBUTING.md.

Writing Dags

Schedule on an event name and read the delivery from triggering_asset_events. Each event's extra holds the payload, or for Jira a summary of it:

from airflow.sdk import Asset, dag, task

deployed = Asset("demo/*/deployed")


@dag(schedule=deployed)
def on_any_deployment():
    @task
    def handle(*, triggering_asset_events=None):
        for event in triggering_asset_events[deployed]:
            print(event.extra["payload"])

    handle()


on_any_deployment()

Iterate over all events rather than taking the first: before Airflow 3.4, and on any Dag with batch_asset_events, deliveries that arrive before the scheduler creates a run share that run. Events are not ordered; sort by event.timestamp if order matters. extra also holds source, path and delivery_id (where the service sends one). Anyone who can read asset events can read the payload too; see Security.

More examples, for GitHub labels and pushes, organization-wide releases and GitLab pipelines, are in examples/dags/ci_cd.py, and for Jira transitions and new issues in examples/dags/jira_issues.py.

Reference

Connection

One connection per webhook, named webhook_<name>, where <name> is lowercase letters, digits, - and _. Use the connection type Inbound Webhook (inbound_webhook) to get a form in the Airflow UI; its fields are stored in the connection extra under the keys below. The plugin reads connections of any type, so connections created elsewhere, for example from environment variables, work too. Fields left empty are treated as unset.

Connections stored in Airflow's metadata database, in environment variables (AIRFLOW_CONN_WEBHOOK_<NAME>) or in a secrets backend all work.

Configuration

Options in the [inbound_webhooks] section of airflow.cfg on the API server, or the environment variables AIRFLOW__INBOUND_WEBHOOKS__<OPTION>:

Responses and troubleshooting

Security

• The endpoint is open to anyone who can reach it. It does not use Airflow's login: it has to accept services that cannot log in. Every delivery is verified against the webhook's secret before anything is recorded, and unknown webhook names are answered with 404.
• Only listed webhooks reach your secrets backend. A request for a name that is not in webhook_names is refused before any connection is read, and each listed webhook's connection is read at most once per connection_cache_ttl by each API server worker, whatever arrives. So a flood of unauthenticated requests costs your secrets backend about one call per webhook a minute. Setting webhook_names to * gives this up.
• Expose only your webhooks. If the API server must be reachable by an external service, put a reverse proxy in front that forwards only their paths, such as /webhooks/github, rather than exposing the whole Airflow UI and API, and have it rate-limit them.
• Use HTTPS. It matters most for token_header and GitLab, where the secret itself travels in a header.
• Use strong secrets, ideally from a secrets backend, which keeps them out of the metadata database. After rotating one, deliveries signed with the new secret are refused until the cached connection expires (connection_cache_ttl); redeliver them from the service's delivery log.
• Payloads are visible to anyone who can read asset events. Each asset event stores its delivery's payload, and the Airflow UI and REST API show it to every user allowed to read assets, not only to the Dags that subscribe. GitHub push payloads, for example, include each commit author's name and email address. jira stores only a summary of the issue by default.
• No replay protection yet. A captured delivery can be sent again, also with headers its signature does not cover changed. Where the event name comes from such a header, as GitHub's X-GitHub-Event or an event_header does, a replayed body can be given another event's name. Signatures over a timestamp would allow rejecting old deliveries; see docs/design.md.
• Bodies larger than max_body_bytes are refused before verification.

To report a vulnerability, see SECURITY.md.

Compatibility

CI tests the built wheel against Airflow 3.1.1, the newest patch of each supported minor version and the newest release, each on the oldest and the newest Python it supports (3.10 to 3.13, and 3.14 from Airflow 3.2) with SQLite, and against 3.1.1 and the newest release on PostgreSQL 17 and MySQL 8.4. Each test run starts the real airflow api-server and sends it signed deliveries.

Versioning

From 0.1.0 the project follows Semantic Versioning for its public interfaces:

• the endpoint POST /webhooks/<name>: its status codes, and the status, asset_events and candidates fields of its responses;
• the inbound_webhook connection type, the connection options of each source type and the [inbound_webhooks] configuration options;
• event names;
• each asset event's extra: the keys source, path, delivery_id and payload, and for jira the issue fields its summary keeps.

Below 1.0, a change that breaks any of them ships only in a new minor version, and the changelog describes it. The Python modules are not a public interface yet: code that imports from airflow_inbound_webhooks can break in any release. Neither are the messages in a response's detail or in the API server log.

Limitations

• No replay protection.
• Each recorded asset event stores the full delivery payload, except that jira stores a summary by default, and anyone who can read asset events can see it.
• Asset events from webhooks are not written to Airflow's audit log, because a delivery has no Airflow user.
• Renaming a repository, project or service changes its event names, so Dags must be updated.
• The plugin uses some Airflow internals (asset_manager, AssetModel, create_session), which CI checks against every supported version.

docs/design.md covers the design, these limitations and the path towards Airflow core.

FAQ

Why not call Airflow's REST API from the service? Most services can only send webhooks; they cannot obtain an Airflow token or post Airflow's request format. If a service can run arbitrary code, such as a CI job, the REST API (POST /api/v2/assets/events) is the simpler choice.

How does this compare to event-driven scheduling with AssetWatcher? AssetWatcher pulls events from a queue such as SQS or Kafka. The queue keeps events while Airflow is down, so it suits high-volume or must-not-lose streams. Webhooks are pushed: simpler to set up, but a delivery made while the API server is unreachable is only retried if the sending service retries.

Does it replace sensors? For "wait until another system finishes something", yes: the other system's webhook triggers the Dag instead of a sensor polling it.

Contributing

See CONTRIBUTING.md for development setup, tests and adding a source type, and CHANGELOG.md for changes.

License

Apache License 2.0.

Apache Airflow, Airflow, Apache, the Apache feather logo, and the Apache Airflow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. This project is not affiliated with or endorsed by The Apache Software Foundation.