dte — map it once, run it forever

Beta. Interfaces and the dte schema may change between releases. Don't run it on production data without backups. Issues welcome; PRs from approved collaborators only (see CONTRIBUTING).

Your domain expert maps the source — once, click by click. Airflow runs it every day after.

dte is a batch data transformation engine that runs as an Airflow 3 app: a visual pipeline builder on top of a typed backend (DuckDB + ibis + dlt) that turns hand-drawn field mappings into reproducible runs. Every pipeline is an Airflow DAG; Airflow owns login, connections, scheduling and run history.

Quickstart

DTE is an Airflow plugin. Install it into the Airflow 3.3+ you already run, with Airflow's own constraints file, unmodified:

AIRFLOW=3.3.2 PY=3.12
curl -fsSLo constraints.txt \
  https://raw.githubusercontent.com/apache/airflow/constraints-$AIRFLOW/constraints-$PY.txt
pip install dte-airflow --constraint constraints.txt
export DTE_DATABASE_URL=postgresql://user:pass@host:5432/dte   
dte db upgrade

Then drop the one-line DAG file into your DAGs folder and restart Airflow. Full steps, extras and environment: Install into your own Airflow.

DTE is under Plugins ▸ DTE Pipelines in Airflow's left nav (Transform, Quality Check); sources, destinations and lookup tables are under Admin. Eval and Visualize pipelines are API-only for now.

1. Use any Airflow connection whose provider is installed (Admin → Connections, AIRFLOW_CONN_* variables or a secrets backend). See Connections.
2. Under Admin, create a DTE source and destination that use it; then create a pipeline in DTE Pipelines.
3. Click Run. The run is a dag run of dte__<pipeline id>, visible in both UIs.

No Airflow yet? Try it with Docker Compose

git clone https://github.com/flowexperts/dte-airflow.git
cd dte-airflow
scripts/bootstrap.sh          # writes .env: fresh secrets + admin password
docker compose up -d --build

Open http://localhost:8080 and log in as admin with the password bootstrap.sh printed (it's also in .env). The stack listens on localhost only and is meant for local use, not a server.

Ports:

• 8080 — Airflow (UI, REST API, DTE at /dte)
• 5432 — postgres (Airflow metadata; DTE tables in the dte schema)
• 5433 — demo warehouse

What's inside

• backend/ — Python package dte (FastAPI API, ibis compiler, DuckDB engine, alembic migrations, tests).
• frontend/ — React app built by Vite into an Airflow react_apps bundle; renders natively inside Airflow at /plugin/dte with Airflow's theme.
• backend/dte/airflow/ — the Airflow integration: plugin, DAG factory, operator, connection adapter.
• dags/dte_dags.py — one line that turns every DTE pipeline and schedule into a DAG.
• plans/ — numbered feature plans documenting the architecture and each shipped change.

Testing

Three tiers — run only what the change touches:

Airflow tier

scripts/test-pip-install.sh builds the wheel, installs it into a stock apache/airflow image with Airflow's constraints file unmodified, and runs pipelines through Airflow, each through its provider: MinIO (S3, amazon), Azurite (Azure, microsoft-azure), fake-gcs-server (GCS, google), an SFTP server (sftp) and Postgres (postgres). See Install into your own Airflow.

Features

Legend: Available — shipped and working · Hidden — implemented, off by default (feature flag) · Planned — stubbed or in progress

Sources

Destinations

Transform Steps

Connections

DTE uses Airflow connections and the provider that owns them does the auth, so IAM roles, service principals, managed identities and service-account keys work as they do in your other DAGs. A connection type shows up in DTE only when its provider is installed; otherwise the picker names the package to install.

Connections resolve like any task's: secrets backends, AIRFLOW_CONN_* variables, then the metadata DB. Test calls the provider's own connection test (follows Airflow's [core] test_connection).

Runtime & Pipeline Features

Engines

Auth & Access Control

Docs

Full documentation: dte.flowexperts.ai/docs. The source lives in docs/user/; edits there publish on merge to main.

• User guide — start here if you're using the app.
• Install into your Airflow — the pip install.
• Try it with Docker Compose — local stack.
• Architecture — how it fits together.

Built on

DTE is a thin layer over open-source projects other people build and maintain:

• Apache Airflow schedules, runs and records every pipeline.
• DuckDB executes each step, in process, on the Airflow worker.
• ibis compiles steps to SQL and pushes work down to your database.
• sqlglot translates SQL between dialects.
• dlt loads REST APIs and MongoDB, and writes to SQL destinations.

Every project and its license: https://dte.flowexperts.ai/open-source/

Contributing and security

See CONTRIBUTING.md, SECURITY.md and the Code of Conduct.

License

Apache-2.0 — see LICENSE.

Apache Airflow, Airflow and the Airflow logo are trademarks of the Apache Software Foundation. DTE is an independent project, not affiliated with or endorsed by the ASF.