etlantic 0.11.0

Typed, contract-driven data pipeline modeling for Python.

ETLantic

Catch incompatible data-pipeline wiring before you process data.

Define datasets, transformations, and pipelines as typed Python classes. Validate and plan them once. Run locally today with registered implementations; optional Polars, Pandas, SQL, PySpark, and Airflow plugins attach when installed. Portable @Transformation.portable authoring ships in 0.11; portable compilers remain 0.12–0.15.

Status: Alpha 0.11.0 — local runtime + optional Polars/Pandas/SQL/PySpark/Airflow plugins, plus portable @Transformation.portable authoring to dtcs.transform-plan/2. Structured Streaming is experimental. Portable compilers remain planned for 0.12–0.15.

Install

Requires Python 3.11+.

pip install etlantic
etlantic --version

This is everything required for the local quickstart. Add an engine only after the first successful run: choose an integration.

From source

Requires uv.

git clone https://github.com/eddiethedean/etlantic.git
cd etlantic
uv sync
uv run python -c "import etlantic; print(etlantic.__version__)"
uv run python examples/quickstart.py

Quick example

from etlantic import (
    Data,
    Input,
    Output,
    Pipeline,
    PipelineRuntime,
    Sink,
    Source,
    Transformation,
)


class RawCustomer(Data):
    customer_id: int
    first_name: str
    last_name: str


class Customer(Data):
    customer_id: int
    full_name: str


class NormalizeCustomers(Transformation):
    customers: Input[RawCustomer]
    result: Output[Customer]


class CustomerPipeline(Pipeline):
    raw: Source[RawCustomer] = Source(binding="customer_source")
    normalized = NormalizeCustomers.step(customers=raw)
    curated: Sink[Customer] = Sink(
        input=normalized.result,
        binding="customer_sink",
    )


@NormalizeCustomers.implementation("local")
def normalize(customers: list[RawCustomer]) -> list[Customer]:
    return [
        Customer(
            customer_id=row.customer_id,
            full_name=f"{row.first_name} {row.last_name}",
        )
        for row in customers
    ]


CustomerPipeline.validate(profile="development").raise_for_errors()

runtime = PipelineRuntime()
runtime.memory.seed(
    "customer_source",
    [RawCustomer(customer_id=1, first_name="Ada", last_name="Lovelace")],
)
run_report = CustomerPipeline.run(profile="development", runtime=runtime)
print(runtime.memory.get("customer_sink"))

Catch bad wiring before processing data—change the sink type and validate() fails with a structured diagnostic instead of a runtime surprise.

Run the complete tested version at examples/quickstart.py.

Current capability boundary

Capability	0.11
Typed modeling, validation, contracts, and planning	Available
Local Python execution and run reports	Available
Memory, callable, JSON, CSV, and no-write storage	Available
Polars and Pandas dataframe plugins	Available (etlantic-polars / etlantic-pandas)
SQL plugin	Available (etlantic-sql)
PySpark plugin + local provider	Available (etlantic-pyspark)
Structured Streaming	Experimental
Airflow orchestrator compiler	Available (etlantic-airflow)
CLI compile / generate / schema / SARIF	Available
Plugin allowlists / keyring / SQLModel extras	Available
SparkForge migration adapter	Available (etlantic-sparkforge)

Next design line: Portable authoring ships in 0.11. Releases 0.12–0.15 add Polars, PySpark, Pandas, and safe SQL compilers for the same dtcs.transform-plan/2 plans. See the portable transformation guide and roadmap.

Documentation

Hosted docs: etlantic.readthedocs.io

• Getting Started (start here)
• Current 0.11 User Guide
• Quickstart
• Capabilities and Limitations
• Evaluator brief
• Core Concepts
• Architecture
• Contributing
• Roadmap

Build the docs locally with uv run mkdocs serve.

Development

Requires uv.

uv sync
uv run pytest
uv run ruff check .
uv run ruff format .

uv sync creates .venv, installs the package in editable mode, and installs the dev dependency group (pytest, ruff, mkdocs) by default.

License

MIT