Skip to main content

airflow-provider-skaledata

SkaleData's Airflow extensions. Pre-installed in ghcr.io/skaledata/airflow — if you're using the SkaleData base image you don't need to do anything.

If you're running your own Airflow image:

pip install airflow-provider-skaledata

The PyPI distribution is airflow-provider-skaledata (matches the Apache community-provider naming convention). Imports live under the skale.providers.* namespace.

Namespace

Mirrors Airflow's own provider layout (airflow.providers.<name>.{hooks,operators,triggers}.<name>) under skale.providers.*:

from skale.providers.airbyte.hooks.airbyte import AirbyteHook
from skale.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator
from skale.providers.airbyte.sensors.airbyte import AirbyteJobSensor
from skale.providers.airbyte.triggers.airbyte import AirbyteSyncTrigger

Subsequent SkaleData providers (skale.providers.<other>...) slot in here.

What's in it today

skale.providers.airbyte — bearer-auth shim for managed Airbyte

Drop-in replacements for the upstream Airbyte provider's hook, operator, sensor, and trigger that authenticate via a static bearer token (your SkaleData sdk_* API key) instead of the upstream OAuth2 /applications/token flow.

SkaleData ships its managed Airbyte with global.auth.enabled: false. The Caddy ingress in front of Airbyte validates the API key at the edge, so the standard apache-airflow-providers-airbyte connector — which only supports OAuth2 client credentials or no-auth — can't talk to it.

Airflow connection setup

Field Value
Conn Type Airbyte
Host https://<cluster>.skaledata.run/api/public/v1/
Password (client secret) sdk_... (your SkaleData API key)
Login / Token URL leave blank
Extra (optional) {"timeout": 60} — seconds per API request; defaults to 30 (airbyte-api 1.x)

DAG usage

from datetime import datetime
from airflow.decorators import dag
from skale.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator


@dag(start_date=datetime(2026, 1, 1), schedule=None, catchup=False)
def run_airbyte_sync():
    AirbyteTriggerSyncOperator(
        task_id="sync_postgres_to_warehouse",
        airbyte_conn_id="airbyte_default",
        connection_id="<your-airbyte-connection-uuid>",
        deferrable=True,
    )


run_airbyte_sync()

AirbyteTriggerSyncOperator is a drop-in for the upstream AirbyteTriggerSyncOperator — same arguments, same async/deferrable semantics.

Two differences from upstream make retries safe:

  • Running syncs are reused. If Airbyte rejects the submit with a 409 because a sync is already running for the connection (for example, a retry of a try that submitted one), the operator attaches to that job instead of failing.
  • Status polls tolerate transient errors. AirbyteSyncTrigger keeps polling through up to two consecutive failed status requests (a timeout, a 5xx) and fails only on the third.

Asynchronous (idempotent) usage

With asynchronous=True the operator returns the Airbyte job id immediately and AirbyteJobSensor waits on it as a separate task. Because the submit and the wait are split, a sensor retry re-checks the existing job instead of kicking off a duplicate sync — mirroring the upstream async example:

from datetime import datetime
from airflow.decorators import dag
from skale.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator
from skale.providers.airbyte.sensors.airbyte import AirbyteJobSensor


@dag(start_date=datetime(2026, 1, 1), schedule=None, catchup=False)
def run_airbyte_sync_async():
    trigger_sync = AirbyteTriggerSyncOperator(
        task_id="trigger_airbyte_sync",
        airbyte_conn_id="airbyte_default",
        connection_id="<your-airbyte-connection-uuid>",
        asynchronous=True,
    )
    wait_for_sync = AirbyteJobSensor(
        task_id="wait_for_airbyte_sync",
        airbyte_conn_id="airbyte_default",
        airbyte_job_id=trigger_sync.output,
        deferrable=True,
    )
    trigger_sync >> wait_for_sync


run_airbyte_sync_async()

AirbyteJobSensor supports both poke mode (default) and deferrable=True, same as upstream.

Metadata

Release files for airflow-provider-skaledata 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for airflow-provider-skaledata 0.8.0
File Size Uploaded Provenance Actions
airflow_provider_skaledata-0.8.0.tar.gz 24.2 kB With provenance Details for airflow_provider_skaledata-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for airflow-provider-skaledata 0.8.0
File Interpreter ABI Platform Actions
airflow_provider_skaledata-0.8.0-py3-none-any.whl Python 3 none any Details for airflow_provider_skaledata-0.8.0-py3-none-any.whl

Total release size: 42.4 kB

Release files / airflow_provider_skaledata-0.8.0.tar.gz

Download URL airflow_provider_skaledata-0.8.0.tar.gz
Size 24.2 kB
Tags Source
SHA-256 checksum
How to use checksums
0e833191b4bec715b5ca9fd28a29e8ee1f566c7d2c7ecf4bf1535f458e12ba92
BLAKE2b-256 checksum
How to use checksums
17a64b159aa030049be38a1c0f5046ed71391babc484d540284e46094d9d6b67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / airflow_provider_skaledata-0.8.0-py3-none-any.whl

Download URL airflow_provider_skaledata-0.8.0-py3-none-any.whl
Size 18.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1b00b398683731a91c196dd971fcbd9c9f4adc8213156acf8b3d269d73678187
BLAKE2b-256 checksum
How to use checksums
2c81be0fe461ef4fc877139f0a4bf5926ca35da2f513ac2d6b6e6758c7e00b0e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for airflow-provider-skaledata 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for airflow_provider_skaledata-0.8.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page