ETLantic

Typed Python data pipelines with validate-before-write.
Design once. Validate everywhere.

Documentation · Quickstart · Compare · Capabilities

ETLantic lets you define Python data pipelines as typed classes or functional builders / JSON (PipelineDefinition), catch bad wiring and contract mismatches before any write, then run or compile the same pipeline on local Python, Polars, Pandas, SQL, or Spark—and emit Airflow DAGs when you need them.

It is not a warehouse tool (use dbt), not a scheduler (use Airflow, Dagster, or Prefect), and not a dataframe engine. It is a typed pipeline framework that coordinates the tools you already choose.

Typed contracts ──▶ Validation ──▶ Deterministic plan ──▶ Run or compile

Not sure if ETLantic fits? Start with Compare.

Why ETLantic?

• Catch invalid wiring, incompatible contracts, missing capabilities, and untrusted plugins before a write.
• Validate extracted inputs, transformation outputs, engine transitions, and publication boundaries against the same contracts.
• Keep one logical pipeline across local Python, Polars, Pandas, SQL, and PySpark; compile to Airflow DAGs; run under Prefect where the local MVP applies.
• Review deterministic, secret-free plans and preserve structured diagnostics, lineage, schema observations, and run reports.
• Install a small core and add only the engines you need.

Quickstart (start here)

Primary path: CLI init → validate → run (file-backed sample). Requires Python 3.11+. Use an empty directory for init (or pass --force).

pip install etlantic
python -m etlantic --version

mkdir my-pipeline && cd my-pipeline
python -m etlantic init --with-toml
python -m etlantic validate pipeline.py:SamplePipeline --profile development
python -m etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json

You should see run status succeeded and JSON rows for Ada and Grace (identity transform on the sample). That proves plumbing—next, change the transform in First Pipeline.

The CLI defaults to development when --profile is omitted (or your project's default_profile). Prefer an explicit profile in scripts and CI.

Full walkthrough: Quickstart.

After first success (clone only): repository demos under examples/ require a git checkout — they are not in the PyPI wheel. Pip-only users: ignore examples/ until you clone.

Status: ETLantic is currently Beta and suitable for documented single-tenant pilots—not unrestricted enterprise production. Structured Streaming remains experimental. See Capabilities and Production readiness.

Engines and integrations

See Optional packages for observability (otel / observability extras) and Arrow helpers. Official engine packages share the 0.34 Beta pilot envelope even when PyPI classifiers say Stable—treat the docs narrative as authoritative.

Matching extras such as etlantic[polars] are equivalent. Pin matching minors while ETLantic follows its 0.x roadmap.

After Ada/Grace — SDK sketch

Once the CLI Quickstart succeeds, the same model fits in a few lines of Python (memory-backed demo; seed data yourself):

import etlantic as etl


class RawCustomer(etl.Data):
    customer_id: int
    first_name: str
    last_name: str


class Customer(etl.Data):
    customer_id: int
    full_name: str


class NormalizeCustomers(etl.Transformation):
    customers: etl.Input[RawCustomer]
    result: etl.Output[Customer]


@NormalizeCustomers.implementation("local")
def normalize(customers: list[RawCustomer]) -> list[Customer]:
    return [
        Customer(
            customer_id=row.customer_id,
            full_name=f"{row.first_name} {row.last_name}",
        )
        for row in customers
    ]


class CustomerPipeline(etl.Pipeline):
    raw: etl.Extract[RawCustomer] = etl.Extract(asset="customers")
    normalized = NormalizeCustomers.step(customers=raw)
    output: etl.Load[Customer] = etl.Load(
        input=normalized.result,
        asset="normalized_customers",
    )


profile = etl.Profile(
    name="demo",
    assets={"customers": "memory", "normalized_customers": "memory"},
)
runtime = etl.PipelineRuntime()
runtime.memory.seed(
    "customers",
    [RawCustomer(customer_id=1, first_name="Ada", last_name="Lovelace")],
)

CustomerPipeline.validate(profile=profile).raise_for_errors()
plan = CustomerPipeline.plan(profile=profile)
run = CustomerPipeline.run(profile=profile, runtime=runtime)

Longer SDK walkthrough: SDK 10 minutes (after CLI first success).

Contract artifacts

Your Python types are also portable, reviewable contract artifacts. Generate the complete bundle from a valid pipeline:

python -m etlantic generate pipeline.py:SamplePipeline -o contracts/

contracts/
├── data/              # ODCS data contracts
├── transformations/   # DTCS transformation contracts
└── pipelines/         # DPCS pipeline contract

Generation is deterministic and refuses invalid pipelines, so contract changes can be reviewed and versioned alongside the code that defines them.

Architecture

ETLantic keeps logical meaning separate from physical execution:

Data + Transformation + Pipeline contracts
                              │
                       validate and plan
                              ▼
                    secret-free PipelinePlan
                              │
                  ┌───────────┼───────────┐
                  ▼           ▼           ▼
               execute      compile     generate
                  │           │           │
                  └──── plugins and external systems

Plans and reports contain secret references, never resolved secret values. Production profiles require explicit plugin allowlists. Backend optimizations may change the physical graph but must preserve contracts, validation boundaries, security domains, and logical attribution.

Interchange formats and the validation envelope are covered in Architecture and Validation Everywhere.

Capability boundary

Full matrix: Capabilities. Roadmap programs live under docs Contribute → Maintainers (for example the multi-tenant control-plane plan) — not day-0 reading.

Learn more

Installation · Quickstart · Compare · Engine selection · Security · Roadmap · Contributing

MIT licensed.